Nu pot deschide un fisier ca are dimensiunea de 100 Gb, Scoup este a fi filtrat ca dupa sa ajunga intr-o baza de date, insa cel mai probabil vor fi ceva filtre dupa anumite caractere folosind find and replace fie ide ul, fie un csv format.
Initial am incercat sa deschid cu sublime chiar si cu php storm, insa a dat fail, ca si alternativa in minutul doi ma gandeam sa folosesc “grep ul de la linux” daca nu ma insel pentru a scrie o comanda de divizare per 1 gb sau 10 gb, insa ma intreb daca exista vrun program sau o solutie mai buna, din nou fac apel la experienta si inceleptiunea voastra,multumesc!
Serios? Chiar voiai să deschizi un fişier de 100GB cu un editor de text, care incearca sa incarce tot fisierul in RAM? Dă si tu mai multe detalii, ce fisier e ala, text? Il poti parsa linie cu linie? Sau…?
A fost doar un click, nu ma asteptam sincer,eram curios doar de limita si continut efectiv. Spre exemplu se pare ca o linie are in jur de 100-200 caractere iar in php storm cand trece de 2.5 mb se opreste.
Fisierul intial este de tip text, insa nu are extensie , deci ii pot face rename si pune ce extensie vreau.
Daca este fisier text si vrei sa faci find & replace prin el, poti folosi un “stream-editor”, sed. Daca vrei sa il procesezi prin metode proprii, poti sa foloseşti mmap (2) sa mapezi tot fisierul in memoria virtuala şi să te joci cu el ca si cum ar fi un imens array.
EmEditor easily opens up to 248 GB or 2.1 billion lines – whichever is reached first, when opening a file – with only a little memory. For files even larger than that, use the built-in Large File Controller.
EmEditor is built to agilely handle files of any size. When you ask it to open a file over a certain size (you choose the size), EmEditor will automatically start using temporary disk space rather than clogging up your memory, unlike most other text editors, which try to keep the whole file in memory and ultimately fail. By default, EmEditor uses temporary files when it opens a file larger than 300 MB. You can check and edit this size in the Advanced tab of the Customize dialog box. If you open a file larger than this size, a few highlighting features are disabled, including multiple-line comments. Wrapping modes are also disabled for optimal speed. If you are opening a file larger than this size, make sure there is enough disk space in the temporary file folder. The default temporary folder is the system temporary folder, specified by the %TEMP% environment variable. You can override the temporary folder to any folder you would like, that has enough space available.
EmEditor’s multithreaded design allows you to view documents during the opening of a large file. A status window appears during most time-consuming activities such as text editing, saving, searching, replacing, inserting and deleting, which allows you to monitor and cancel those activities at any time.
Ma gandesc ca daca vrei sa il separi in componente inserabile intr-o baza de date ai putea face asta cu awk. Asa poti filtra si genera inclusiv instructiunile insert.
Imi amintesc ca am folosit LTF Viewer pentru a deschide fisiere mari.
Nu incarca tot in memorie ci bucati, pe care le poti specifica (de la randul x la randul y).
Dă si tu mai multe detalii, ce fisier e ala, text? Il poti parsa linie cu linie? Sau…?
… astept o ieftinire de ram si sa treaca craciunul, anul nou… am banii calculati pana pe 15 ianurie.