The client is a leading name in Oil and Gas industry. Facing multiple scalability issues, they engaged Simform for a database migration strategy and build a future-proof-scalable database. This database is currently being consumed by an Oil and Gas geolocation software to identify potential locations for Upstream operations.
We had to migrate millions of rows of data from PostGis to MongoDB.
Some records in the data were not satisfactory to build an efficient database. Also, some tables were making it almost impossible to merge the data.
MongoDB doesn’t creates the same indexing as PostGis upon migration. We built reference structures to facilitate migration from RDBMS to NoSQL. Without building reference structures, direct migration of the database would create a corrupted database.
We converted shapefiles into PostGis using qGIS. PostGis stores geographical information into binary, and decrypting for MongoDB is difficult. We then converted this geographical information into plain text, finally, converting it into GeoJSON.
When we looked into various shape files, we found extremely high granularity and multiple duplications in the existing data. To import an error free database, we optimized polygon points in shapefiles to remove any duplication within these records.
We created a simple yet very integrable, high maturity data model. This data model further facilitated building business applications under very restricted resource environments. The solution leverages Google maps with on demand data binding.
Most data tables came from different sources, many had different formats, granularity and indexing. We unified and merged these tables into a single document to create a flexible schema design for MongoDB.
The data was migrated in batches, each batch was evaluated for errors upon successful migration. As the database scales, we wanted to make sure that complexities and growth limitations remain simple. With elastic deployments, we improved business application performance during load spikes.
Before moving to our modern database, client was facing issues while scaling with millions of rows of data. After migration, their business can now consume and process 20 Million+ rows of data on a scale, helping them leverage simplified and high performing data models to power enterprise applications and BI dashboards.