Input and output
I have received a file which was roughly SQL. Here is the sample
- Deleting tabs
- Deleting Empty lines
- Opening brace
- Closing brace
- Clean up
Tab character are usually not displayed in most non-developer focused text-editors. they fall under hidden characters category, this can be removed using s command.
Deleting Empty lines
In, The next stage of our data processing pipeline, we will eliminate all empty line from the file generated in step 1. This is achieved with d command.
Line starting with CREATE should have an opening brace at the end the line.
This was an interesting one, The goal was to mark end of the table definition ie. ); .This was solved by inserting line before line containing CREATE. this uses i command
For the output file to be valid SQL, every column definition must be separated by a comma. The below command does a reverse match, So it matches all lines not starting with CREATE and appends a comma at the end of the line.
This command is handling the undesired byproduct created in above step. The reverse match in previous step matched all lines including the table definition endings. This removes that extra comma
Did I get this working in one attempt? No. It was a series of trial and errors involving refering official manual and some quick searches on the web.
Here are the key takeaways.
- Breaking down problem into small sub-problem will always help
- Avoid using -i flag with sed, this performs in updates to the file, which may not be desirable depending on the end goal.