UPDATE:
As per the answer suggesting the use ofTable::insert
, I updated the query.
// At this point the data is stored in an array
// I turn it into a collection to make use of the chunk function
$records = collect($records);
// I split the array into chunks
// Inserting everything at once gave memory issues
$chunks = $records->chunk(500);
// Then I iterate the chunks and insert them into the database
foreach ($chunks as $chunk) {
\DB::table('table')->insert($chunk->toArray());
}
The issue with this is that while being faster than using the previous method, it still takes several minutes to insert everything into my database, so it's not an actual solution.
I'm not sure if this is the wrong approach, if I'm missing something/doing something wrong here.
Description:
I have an Angular form where the user can upload a .txt file containing CSV data. This is how it works:
HTML:
<input type="file" class="form-control w-auto" (change)="getFile($event)" accept=".txt" />
Typescript:
getFile(e) {
this.file = e.target.files[0];
this.file_description = e.target.files[0].name;
}
submit() {
let reader = new FileReader();
reader.onload = () => {
let result = reader.result;
this.ImportService.import(result).subscribe(
(response) => {
// Do success stuff
},
(_error) => {
// Do error stuff
}
);
};
if (this.file) {
reader.readAsText(this.file, "UTF-16LE");
}
}
This sends the data to my Laravel instance, where I want to store it into MySQL. I've been able to do this by processing the received data into an array and inserting each row one by one, like this:
PHP:
$array = array();
$csv = str_getcsv($request->file, "\n");
foreach ($csv as &$row) {
$row = str_getcsv($row, ";");
$array[] = $row;
}
array_splice($array, 0, 1);
foreach ($array as &$row) {
$query = Table::firstOrNew(['col2' => $row[1], 'col3' => $row[2]]);
$query->col1 = $row[0];
$query->col2 = $row[1];
$query->col3 = $row[2];
$query->col4 = $row[3];
$query->col5 = $row[4];
// [...]
$query->col72 = $row[71];
$query->col73 = $row[72];
$query->save();
}
The Problem:
The data being sent here contains around 100.000 records. This method is way too slow, often leading to timeouts/5-10+ minutes of waiting.
Attempted Solution:
I've been trying to useLOAD DATA LOCAL INFILE
, but I can't get it to work.
Here's the code:
$results = DB::connection()->getpdo()->exec(
"LOAD DATA LOCAL INFILE '" . $request->file . "' IGNORE INTO TABLE `table`
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (
`col1`,
`col2`,
`col3`,
// etc
)"
);
In short, there are three issues with this method:
local_infile
property on my MySQL instance is disabled by default, and reverts to disable on restart.General error: 7890 Can't find file 'col1;col2;col3;etc'
.I've tried usingLOAD DATA INFILE
but I'm getting several 'access denied' errors when the query tries to fetch the file contents.
If you need any further information pleases let me know.
Tasks like this are probably better suited to be handled in a Queue/Job so that the user isn't waiting for this to complete in real time.
Either way, I would recommend refactoring the code though. firstOrNew() will run queries for every single record you process, so it will be a major strain on the database.
The methods insert() and update() support processing many records at a time, so use PHP to preprocess the records so you can utilize those two methods.
Use one database query to get all the existing records and then build two arrays to call insert() and update().
I think you can create a where query like this if you create pairs of values that you want to search on:
Table::where(function($query) use ($pairs) {
foreach ($pairs as $pair) {
$query->where('col2', $pair['col2'])->where('col3', $pair['col3']);
}
});
That will give you the existing records, so then you can loop through all records in PHP to build your two arrays and then run the insert and update queries only one time:
Table::insert($new_records);
Table::update($existing_records);
Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.
Find the answer in similar questions on our website.
Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.
PHP (from the English Hypertext Preprocessor - hypertext preprocessor) is a scripting programming language for developing web applications. Supported by most hosting providers, it is one of the most popular tools for creating dynamic websites.
The PHP scripting language has gained wide popularity due to its processing speed, simplicity, cross-platform, functionality and distribution of source codes under its own license.
https://www.php.net/
Laravel is a free open source PHP framework that came out in 2011. Since then, it has been able to become the framework of choice for web developers. One of the main reasons for this is that Laravel makes it easier, faster, and safer to develop complex web applications than any other framework.
https://laravel.com/
Angular is one of the most powerful JavaScript frameworks out there. Google uses this platform to develop a Single Page Application (SPA). This development environment is known primarily because it provides developers with a better environment for combining JavaScript with HTML and CSS. More than half a million sites like google.com, youtube.com, etc. use Angular.
https://angular.io/
DBMS is a database management system. It is designed to change, search, add and delete information in the database. There are many DBMSs designed for similar purposes with different features. One of the most popular is MySQL.
It is a software tool designed to work with relational SQL databases. It is easy to learn even for site owners who are not professional programmers or administrators. MySQL DBMS also allows you to export and import data, which is convenient when moving large amounts of information.
https://www.mysql.com/
HTML (English "hyper text markup language" - hypertext markup language) is a special markup language that is used to create sites on the Internet.
Browsers understand html perfectly and can interpret it in an understandable way. In general, any page on the site is html-code, which the browser translates into a user-friendly form. By the way, the code of any page is available to everyone.
https://www.w3.org/html/
Welcome to the Q&A site for web developers. Here you can ask a question about the problem you are facing and get answers from other experts. We have created a user-friendly interface so that you can quickly and free of charge ask a question about a web programming problem. We also invite other experts to join our community and help other members who ask questions. In addition, you can use our search for questions with a solution.
Ask about the real problem you are facing. Describe in detail what you are doing and what you want to achieve.
Our goal is to create a strong community in which everyone will support each other. If you find a question and know the answer to it, help others with your knowledge.