{"id":404682,"date":"2022-03-31T20:01:45","date_gmt":"2022-03-31T20:01:45","guid":{"rendered":"https:\/\/uat.esri.com\/en-us\/industries\/blog\/?post_type=blog&p=404682"},"modified":"2024-07-22T19:38:40","modified_gmt":"2024-07-22T19:38:40","slug":"metaphors-matter-data-integration-and-management","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/en-us\/industries\/blog\/articles\/metaphors-matter-data-integration-and-management","title":{"rendered":"Metaphors Matter \u2013 Data Integration and Management"},"content":{"rendered":"
I recently came across a Power Point deck that I used to use to discuss data integration, and the role of GIS within a large organization. At the bottom of the slide were all of the typical key databases within a State Department of Transportation: pavement, bridges, crashes, traffic counts, etc. Each was represented by a database \u201ccylinder\u201d and was meant to represent the silos within most DOTs. And from each cylinder came an upward facing arrow pointing to the center of the slide, which was a large band which represented the ability of GIS to serve as an integration framework to bring all of the data from these disparate databases together, which then could be served out to users across the agency.<\/p>\n\n
It was a powerful slide meant to represent the standard challenge at most State DOTs: how to integrate and manage all of the data, which was located at the individual business units, and make it a corporate asset available across the organization to do analysis, and ultimately to make better decisions. And 10 to 15 years ago, just about every State DOT that I was familiar with struggled with these basic data challenges.<\/p>\n\n
Each year the slide became a little more subtle (and complex) as we gained more experience with services oriented architectures, and understood the complexities of integrating disparate data, often housed in different data formats, and utilizing different linear referencing methods. At the same time, the scope of the data to be integrated increased, as we began to think about not just some of the core datasets but looking more widely across these large organizations.<\/p>\n\n
\n<\/figure>\n<\/div>\n\n
Increasingly we began to see that middle band in terms of an enterprise service bus, and any number of State DOTs began to think about implementing an ESB or similar integration framework for their data. But the ways in which we viewed the role of an enterprise service bus was still very much in line with that original graphic: we were integrating disparate databases, most often still represented by \u201ccylinders.\u201d<\/p>\n\n
And not only did the way in which we integrated data subtly change from year to year, but the way we managed that data also changed. In the world of point to point database integration, data management was more episodic and driven by individual business needs, increasingly we began to think about an enterprise strategy for managing data other than the mainframe. Data marts were one of the first enterprise concepts to emerge, and ultimately these were supplanted by data warehouses of various complexities. Some were designed to update every evening, synchronizing any changes which occurred across the respective databases, while others updated less frequently. <\/p>\n\n
Taking that data warehouse one step further, many State DOTs began to think about data lakes, although these two concepts are designed to address different aspects of data management and are not interchangeable. The larger point is that once implemented – often referred to as a system of record – many IT managers often saw their job as complete, that they had created an enterprise data management strategy that housed much of the agency\u2019s data.<\/p>\n\n
And while this is all a bit simplified, I think it also represents the way many of us thought about data integration and data management. But over the last several years, I have begun to think that these cylinder metaphors actually prevent us from seeing the larger data struggles which happen every day in the typical DOT. And here I think it is most useful to use a different metaphor, that of the project lifecycle, and the data that we create as we move through that project lifecycle.<\/p>\n\n
\n<\/figure>\n<\/div>\n\n
The problems are especially apparent when we focus on the \u201chand-offs\u201d as we move through the project lifecycle: As we move from planning into survey and design, there is an amazing amount of uncoordinated or disconnected data, and even more so as we move into construction and construction management. These are some of the central workflows in our agencies (DOTs), and I don\u2019t think anyone thinks we are managing all of that disparate data effectively. Often going under the larger term \u201cdigital delivery,\u201d there is probably not a single State DOT that has successfully integrated and managed the data created through this process. <\/p>\n\n
And this is precisely the process that our European colleagues are focused on, which they refer to as BIM, but what they really mean is what we would call whole lifecycle information management. <\/p>\n\n
The goal of whole lifecycle information management is to capture the typical inefficiencies and information loses as we move between the various stages of a project. Often, we utilize different software platforms, and we often start over with having to recreate data for use in these different platforms. It is the area under the curve in the illustration above that represents the potential efficiency gains if we take our information systems more seriously from the early stages of a project. <\/p>\n\n
This has been the focus in Europe under the banner of BIM, and the growing requirements that all large government funded projects follow BIM standards and procedures. The UK government has mandated BIM Level 2 Compliance on government projects, as have a large number of other European governments. The European Commission has established task forces to move the procedures forward for EU wide adoption, recognizing the productivity gains to be achieved. In the case of the UK, they were estimating a 20 percent project savings associated with implementing BIM procedures, and while not every project has hit this mark to date, the results have been impressive, nevertheless.<\/p>\n\n
It is when we move from a standard IT view of data integration and data management to thinking about specific workflows, that I think we can identify the day to day data struggles that most DOTs are grappling with. When we focus on specific workflows \u2013 and in this move from planning through engineering and design to project delivery there are a very large number of workflows \u2013 then we can begin to think differently about how we organize and manage data to support the main workflows in our agencies. This is one of the central challenges for current DOT IT Departments.<\/p>\n\n
A final point relates to our current systems of record, whether in data warehouses or some other enterprise data management structure. While creating these data warehouses has consumed considerable time and effort, they did not solve the more fundamental problem that most DOTs face: how do we enable the data consumers in our organizations with the data they need to support their day to day work flows that consume so much of the agency\u2019s time, effort, and resources. <\/p>\n\n
The average data consumer is not equipped to run SQL queries, nor is basic database technology friendly to their more business focused workflows.\u00a0 Typically they need a fairly simple application \u2013 what we call maps and apps \u2013 to perform their daily inspections or maintenance activities, and lacking strong support from IT, they often rely on either paper or Excel spreadsheets, which rarely make it back into our enterprise systems of record.\u00a0 And this is where we at Esri feel that empowering these workers with appropriate tools (the data, maps, and apps) they need, is the next biggest challenge for DOT IT departments, what we call systems of engagement.\u00a0 This is the core work needed to digitally transform<\/a> our organizations, and prepare them for planning, designing, and delivering the transportation networks of the future.<\/p>\n\n
![\"\"](\"https:\/\/uat.esri.com\/en-us\/industries\/blog\/wp-content\/uploads\/2022\/03\/metaphor-blog-image-1.png\")
<\/figure>\n<\/div>\n\n![\"\"](\"https:\/\/uat.esri.com\/en-us\/industries\/blog\/wp-content\/uploads\/2022\/03\/metaphor-blog-image2.jpg\")
<\/figure>\n<\/div>\n\n