[metapost] reading a large array

[Qiong Cai]

Yes, I also figured out this way. I used the 2D array to speedup the
reading. See the comparison below.

Thanks all!


// 1D results

\begin{tabular}{ll}

\toprule

Number of Numbers & Reading Time \\

\midrule

10 & 0.096s \\

100 & 0.096s \\

1K & 0.100s \\

10K & 0.463s \\

100K & 1m23s \\

500K & 30m21 \\

\bottomrule

\end{tabular}

\caption{The running time of readFromFile on my iMAC (3.2GHz, SKL). The
solution

is not scaled after 10K data.}



// 2D results, I used 1K for each subarray

\begin{tabular}{ll}

\toprule

Number of Numbers & Reading Time \\

\midrule

100K & 1.075s \\

500K & 5.434s \\

1M & 14s \\

5M & 6m27s \\

\bottomrule

\end{tabular}

\label{table:fastReadFromFile}

\caption{The running time of fastReadFromFile on my iMAC (3.2GHz, SKL). For
500K data points,

it improves the performance by 335x over readFromFile. However, the
solution is not

scalable after 5M data points. I can further improve it by using
multidimenstional arrays

when my data set size reaches 5M.}






On Fri, Mar 10, 2017 at 10:14 AM, luigi scarso <luigi.scarso at gmail.com>
wrote:

> On Fri, Mar 10, 2017 at 3:58 PM, Hans Hagen <pragma at wxs.nl> wrote:
> > this runs in 3 sec on my machine which is way faster than storing at the
> mp
> > end where at some point you hit limitations in constructing names for
> > variables (keep in mind that a[i] is not an array but a hashed variable
> name
> > constructed from a and i)
> >
> >
>
> We have played a bit, and the point is that
> is better to think to a[] as a tree, and lesser is the depth , the better.
>
> For example given c and foo[]
> one can  use foo[c]  to store s, but
> can also declare  foo[][][][] and say
> foo[c div 1000][c div 100][c div 10][c] := s ;
>
> We don't lose items, because the last index of foo is unique, and
> lookup is better
> (of course we have now 3 operations, which cost).
>
> You can experiment with the following code by Hans:
>
>
> string foo[][][][];
>
>     def set_foo(expr c, s) =
>         foo[c div 1000][c div 100][c div 10][c] := s ;
>     enddef ;
>     def get_foo(expr c) =
>         foo[c div 1000][c div 100][c div 10][c]
>     enddef ;
>
>     string s;
>     numeric c ; c := 0;
>     forever:
>         s := readfrom "foo3.tmp";
>         exitif s = EOF;
>         c := c + 1 ;
>         set_foo(c,s) ;
>     endfor ;
>     save l ; numeric l;
>     for i=1 upto c :
>         l := length(get_foo(c)) ;
>     endfor ;
>
>
>
> --
> luigi
> --
> http://tug.org/metapost/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tug.org/pipermail/metapost/attachments/20170310/a10a1935/attachment-0001.html>