Dump File = text file on the website server that appends incoming data to the text file so that there is 1 file containing all data. I’m using it as the best fail-safe for data that I can think of.
Scenario
You have that new project in front of you and you have to build solutions for the client. You meticulously lay out the work flow diagram for your website. You figure out how to collect and pass data to 5 different systems. You launch the website and wait for those form fills to come rolling in!
And they do! Hundreds of form submissions are arriving from the WordPress website and we are happily counting each one.
Until… the client calls and says we are missing form submissions. How can that be? We are carefully monitoring the client’s CMS system to make sure the leads are reaching their destination. Additionally, we have it zapped to a Google spreadsheet. We are sure of data capture… until we pull an extract from the eCommerce system and see that the number of form fills in the eCommerce system do not match the number of purchases in Google Sheets.
I know what to do! Grab the secured dump file off the server and compare it to the ecommerce system to find any leakage of leads. Seems like common sense. Until you find the dump file missing. How could this have happened? the dump file is the #1 thing outlined in the workflow, the absolute first step in the data processing.
Team Fail
This is a team fail. A fail on the fellow developer who failed to implement the dump file like we outlined. A fail on me, because my hands ain’t broke! I could have checked on that file. A fail for my upper-level director who didn’t even know how important the dump file is.
Lesson Learned
Once of the biggest things I learned was to try and get the file right the first time you implement it. I threw an empty file out there in a race to implementation without considering long term implications:
- Once the log file is implemented on the server, it’s hard to stop it. For example if you need to edit the file, it has to stop writing to the file, so (without a backup process) incoming data would not be captured during the editing process.
- Choose your text delimiter and stick with it. I think I did this right by consistently using the pipe ( | ) in the data to mark the beginning of the next field. This makes the data easier to import into other programs, like Excel.
- Don’t forget to add the field names and delimiters at the top of the file as you launch it. Again, easier importation into other programs.
- When you inevitably need to add a column to the data, add the new field to the end. Now is not the time for re-thinking data hierarchy and what piece of data you want to come first, second… if you do, then your earlier data will never align with the later data. You will resort to a second file at that point.
No matter how many fires there are around the project, nothing is as important as having a backup catch-all like the dump file.
