X

Track changes made to this page

If you find this page useful and would like to be notified of changes made to this page, start by inputting your email below.



Privacy policy
Close this window

powered by ChangeDetection

Difference between revisions of "Pennsylvania Oil and Gas Data Sources"

(Created page with " ==May 2015 Production Update== Data within google drive appears to be uploaded to PA for waste and Production, assuming the data we are interested in is located within th...")
 
 
(16 intermediate revisions by 4 users not shown)
Line 1: Line 1:
+
__TOC__
 +
=Pennsylvania Data Releases=
  
==May 2015 Production Update==
 
  
 +
==Pennsylvania v0.1 Alpha (March 2013)==
 +
Originally called the Oil and Gas Wells Database (O&GWDB), this was the very first build of what would become WellWiki. The project was released internally only and contained data on approximately 130,000 wells gathered from the Pennsylvania Department of Environmental Protection. These data included permits, spuds, and compliance reports (from Jan 1900 to Feb 2013) and production and waste reports (from 2000 to mid-2012). The project team consisted of Joel Gehman, Dror Etzion, and Diego Mastroianni.
  
Data within google drive appears to be uploaded to PA for waste and Production, assuming the data we are interested in is located within the PADEP folder within PA.
 
  
However - I suspect that the data was not quite inserted correctly. It appears that the WELL_API_COUNTY_ID on both was not inserted (under the CSVs, they are called Well Permit #).
+
==Pennsylvania v1.0 (May 2014)==
 +
WellWiki was publicly launched. The launch coincided with the release of Pennsylvania v1.0, which consisted of data from the Pennsylvania Department of Environmental Protection on approximately 150,000 oil and gas wells. The project team included Joel Gehman and Michael Hall. Our technology environment consisted of a single server with a 45GB drive and 4GB RAM. The initial software stack included Ubuntu 12.04.4 LTS, MySQL 5.5.31, Apache 2.22.2, PHP, PHPMyAdmin, Pure-FTPd, and SSH.  
  
I think we could correct this, although I don't think we necessarily need to.
 
  
That said, there is still data on the PA DEP website that needs to be uploaded.
+
==Pennsylvania v1.1 (July 2014)==
 +
This update expanded our coverage of Pennsylvania to encompass approximately 250,000 oil and gas wells.
  
  
 +
==Pennsylvania v2.0 (November 2014)==
 +
This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data through October 2014. With its release, WellWiki’s total coverage reached approximately 878,000 wells, primarily due to the inclusion of expanded production data. The project team consisted of Joel Gehman and Michael Hall. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports: permits, spuds, production, waste, compliance, and well pads.
  
Updating production information. Extra column for WELL_API had to be added, using Well Permit #. Excel formula ="37-"&B2 was used to produce WELL_API values. Well Permit # renamed to WELL_API_COUNTY_ID. Extra columns WELL_COUNTRY set to "United States" and WELL_STATE_CODE set to "PA".
 
  
Needed to specify that lines were delimited by \n instead of auto to get around PHP "invalid comma count error". There was a typo in one cell for a sheet that had a "\" character, which produced another invalid column count error. Also renamed all columns in CSV to columns in SQL db as follows:
+
==Pennsylvania v3.0 (June 2015)==
 +
This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data through May 2015. With its release, WellWiki’s total coverage reached 1.5 million wells. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports: permits, spuds, and compliance. Production data spanned from 1980 (the earliest period available) to March 31, 2015 (the latest period available). Waste data spanned from 1995 (the earliest period available) to March 31, 2015 (the latest period available). However, waste records for 1980-1994, inclusive, were not available for download at the time of this update. As part of this upgrade, we created the Pennsylvania Oil and Gas Data Sources wiki page and began footnoting the sources of all data displayed on WellWiki. The project team consisted of Joel Gehman, Mason Strong and Hailey Musselman.
  
WELL_API,WELL_API_COUNTY_ID,PERIOD_ID,PRODUCTION_INDICATOR,WELL_STATUS,FARM_NAME,WELL_ID,SPUD_DATE,GAS_QUANTITY_MCF,GAS_PRODUCTION_DAYS,CONDENSATE_QUANTITY_BBL,CONDENSATE_PRODUCTION_DAYS,OIL_QUANTITY_BBL,OIL_PRODUCTION_DAYS,AVERAGED,OPERATOR_NAME,operator_id,WELL_COUNTY,WELL_MUNICIPALITY,LATITUDE_DECIMAL,LONGITUDE_DECIMAL,UNCONVENTIONAL,CONFIGURATION,HOME_USE,REPORTING_PERIOD,COMMENT_REASON,COMMENT_TEXT,WELL_COUNTRY,WELL_STATE_CODE
 
  
which needs to be entered on the import screen for PHPmyAdmin to upload correctly for column headers.
+
==Pennsylvania v4.0 (February 2018)==
 +
This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells. The project team consisted of Joel Gehman and Keith Mills.
  
This has been more work than it should have been. There is a CSV module in python if we elect to go for full automation, so this could possibly be done automatically.
 
  
 +
==Pennsylvania v5.0 (June 2019)==
 +
This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data available through April 2019. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports:
  
  
 +
  •  Permits<ref>{{cite web
 +
| title = Permits Issued Detail Report
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Permits_Issued_Detail
 +
| accessdate = 2019-05-21 }}</ref>    - 223,184 records from 1800-01-01 to 2019-05-20
  
In addition to removing the \ character from CSVs, the must also be processed to remove the quotation character " to allow for proper importation. Older yearly CSV files (non-monthly) were split using a CSV split utility in order to meet MySQL's 2048KiB filesize limit
+
  •  Spuds<ref>{{cite web
 +
| title = Spud Data Report
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Spud_External_Data
 +
| accessdate = 2019-05-16 }}</ref>      - 213,664 records from 1800-01-01 to 2019-04-30
 +
 
 +
  •  Production<ref>{{cite web
 +
| title =Oil and Gas Production Reports
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?%2fOil_Gas%2fOil_Gas_Well_Production
 +
| accessdate = 2019-05-21 }}</ref>  - 2,305,539 records relating to 148,847 unique wells from 1980-01-01 to 2019-03-31.
 +
 
 +
  •  Waste<ref>{{cite web
 +
| title =Oil and Gas Waste Reports
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?%2fOil_Gas%2fOil_Gas_Well_Waste
 +
| accessdate = 2019-05-21 }}</ref>      - 1,270,740 records relating to 89,734 unique wells from 1980-01-01 to 2019-03-31.
 +
 
 +
  •  Compliance<ref>{{cite web
 +
| title =Oil and Gas Compliance Report
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/OG_Compliance
 +
| accessdate = 2019-05-21 }}</ref>  - 452,647 records relating to 122,490 unique wells from 1950-01-01 to 2019-05-20.
 +
 
 +
  •  Well Pads<ref>{{cite web
 +
| title = Oil and Gas Well Pad Report
 +
| url = http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Well_Pads
 +
| accessdate = 2019-05-21 }}</ref>  - 13,095 records from 1800-01-01 to 2019-05-20.
 +
 
 +
 
 +
Overall, this upgrade resulted in the generation of {{PAGESINCATEGORY:Pennsylvania_Wells}} new or modified Pennsylvania Well Pages; {{PAGESINCATEGORY:Pennsylvania_Operators}} new or modified Pennsylvania Operator Pages; 1,115 new or modified Pennsylvania County and Municipality Pages; and {{PAGESINCATEGORY:Pennsylvania_Well_Pads}} new or modified Pennsylvania Well Pad Pages. The team for this project consisted of Joel Gehman, Cole Miller and Ankesh Kumar.
 +
 
 +
 
 +
==Pennsylvania v6.0 (July 2019)==
 +
In terms of its explicit data contents, this update utilized the same data underlying our v5.0 release. However, whereas all prior Pennsylvania releases utilized our original database design, we built Pennsylvania v6.0 on an expanded version of the database design that accompanied our release of Alberta v2.0<ref>{{cite web | title = Alberta v2.0 Oil and Gas Data Sources | url = http://wellwiki-new-uat.bus.ualberta.ca/wiki/Alberta_Oil_and_Gas_Data_Sources#Alberta_v2.0_.28December_2018.29}}</ref>. Incorporating the Pennsylvania data into this database schema necessitated several enhancements. Notably, the Pennsylvania Department of Environmental Protection provides data on “Well Pads,” basically sites where more than one well may be permitted from the same location. To accommodate these data, we incorporated a new Well Pads table into our database schema. Additionally, the Pennsylvania Department of Environmental Protection reports extensive data regarding inspections, violations, and fines. Comparable data are not available in Alberta, the jurisdiction that served as the basis for our original development of the new database schema. Accordingly, we added new tables related to Inspections, Violations, and Penalties.
 +
 
 +
 
 +
We also made several enhancements to our display and reporting of Pennsylvania oil and gas data on WellWiki. First, Pennsylvania has changed its production and waste reporting requirements for unconventional several times, from annual to semiannual to monthly. Additionally, during this transition there was an odd reporting period that spanned from July 2009 to June 2010. Although it is a crude approximation, for reporting purposes on WellWiki’s production table, we divided the values for this period in half, assigning them equally to two new periods we created: 2009-2 and 2010-1. We then aggregated all available data into annual periods, plus reported any data available for years not yet completed. For instance, if well reported production from 2000-0 to 2019-03, meaning for the full calendar year 2000, through to March 2019, we would report annual values for the 2000 to 2018 period, plus monthly values (if any) for the January to March 2019 period. Our Production table now also includes information on the well operator at the time the production was reported. This makes it apparent whether a well has been bought and sold during its operating history.
 +
 
 +
 
 +
We also expanded the data displayed on our Operator, and County and Municipality Pages, notably to include “operator” on County and Municipality Pages and "County" and “Municipality” on Operator Pages. This mirrors an enhancement we made as part of our Alberta v3.0<ref>{{cite web | title = Alberta v3.0 Oil and Gas Data Sources | url = http://wellwiki-new-uat.bus.ualberta.ca/wiki/Alberta_Oil_and_Gas_Data_Sources#Alberta_v3.0_.28May_2019.29}}</ref> update. Finally, this release incorporated data from our Fracking Chemicals v2.0 update. In total, 8,307 Pennsylvania wells reported data through FracFocus; we include those data for each well. Data on Fracking Activities are obtained primarily from FracFocus. For complete details see the Fracking Activities Data Sources<ref>{{cite web | title = Fracking Activities Data Sources | url = http://wellwiki-new-uat.bus.ualberta.ca/wiki/Fracking_Activities_Data_Sources}}</ref> page.
 +
 
 +
 
 +
Overall, this upgrade resulted in the generation of {{PAGESINCATEGORY:Pennsylvania_Wells}} new or modified Pennsylvania Well Pages; {{PAGESINCATEGORY:Pennsylvania_Operators}} new or modified Pennsylvania Operator Pages; 1,115 new or modified Pennsylvania County and Municipality Pages; and {{PAGESINCATEGORY:Pennsylvania_Well_Pads}} new or modified Pennsylvania Well Pad Pages. The team for this project consisted of Joel Gehman, Cole Miller and Ankesh Kumar. Funding for the project was provided in part by financial support from the Social Sciences and Humanities Research Council of Canada and the University of Alberta, as well as in-kind support University of Alberta School of Business.
 +
 
 +
 
 +
==References==
 +
<references/>
 +
__NOEDITSECTION__
 +
 
 +
[[Category:Documentation]]

Latest revision as of 14:51, 29 August 2019

Pennsylvania Data Releases

Pennsylvania v0.1 Alpha (March 2013)

Originally called the Oil and Gas Wells Database (O&GWDB), this was the very first build of what would become WellWiki. The project was released internally only and contained data on approximately 130,000 wells gathered from the Pennsylvania Department of Environmental Protection. These data included permits, spuds, and compliance reports (from Jan 1900 to Feb 2013) and production and waste reports (from 2000 to mid-2012). The project team consisted of Joel Gehman, Dror Etzion, and Diego Mastroianni.


Pennsylvania v1.0 (May 2014)

WellWiki was publicly launched. The launch coincided with the release of Pennsylvania v1.0, which consisted of data from the Pennsylvania Department of Environmental Protection on approximately 150,000 oil and gas wells. The project team included Joel Gehman and Michael Hall. Our technology environment consisted of a single server with a 45GB drive and 4GB RAM. The initial software stack included Ubuntu 12.04.4 LTS, MySQL 5.5.31, Apache 2.22.2, PHP, PHPMyAdmin, Pure-FTPd, and SSH.


Pennsylvania v1.1 (July 2014)

This update expanded our coverage of Pennsylvania to encompass approximately 250,000 oil and gas wells.


Pennsylvania v2.0 (November 2014)

This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data through October 2014. With its release, WellWiki’s total coverage reached approximately 878,000 wells, primarily due to the inclusion of expanded production data. The project team consisted of Joel Gehman and Michael Hall. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports: permits, spuds, production, waste, compliance, and well pads.


Pennsylvania v3.0 (June 2015)

This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data through May 2015. With its release, WellWiki’s total coverage reached 1.5 million wells. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports: permits, spuds, and compliance. Production data spanned from 1980 (the earliest period available) to March 31, 2015 (the latest period available). Waste data spanned from 1995 (the earliest period available) to March 31, 2015 (the latest period available). However, waste records for 1980-1994, inclusive, were not available for download at the time of this update. As part of this upgrade, we created the Pennsylvania Oil and Gas Data Sources wiki page and began footnoting the sources of all data displayed on WellWiki. The project team consisted of Joel Gehman, Mason Strong and Hailey Musselman.


Pennsylvania v4.0 (February 2018)

This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells. The project team consisted of Joel Gehman and Keith Mills.


Pennsylvania v5.0 (June 2019)

This update expanded WellWiki’s coverage of Pennsylvania oil and gas wells to include data available through April 2019. Data for this update were obtained from the Pennsylvania Department of Environmental Protection and included the following reports:


 •  Permits[1]     - 223,184 records from 1800-01-01 to 2019-05-20
 •  Spuds[2]       - 213,664 records from 1800-01-01 to 2019-04-30
 •  Production[3]  - 2,305,539 records relating to 148,847 unique wells from 1980-01-01 to 2019-03-31.
 •  Waste[4]       - 1,270,740 records relating to 89,734 unique wells from 1980-01-01 to 2019-03-31.
 •  Compliance[5]  - 452,647 records relating to 122,490 unique wells from 1950-01-01 to 2019-05-20.
 •  Well Pads[6]   - 13,095 records from 1800-01-01 to 2019-05-20.


Overall, this upgrade resulted in the generation of 270,818 new or modified Pennsylvania Well Pages; 9,002 new or modified Pennsylvania Operator Pages; 1,115 new or modified Pennsylvania County and Municipality Pages; and 3,798 new or modified Pennsylvania Well Pad Pages. The team for this project consisted of Joel Gehman, Cole Miller and Ankesh Kumar.


Pennsylvania v6.0 (July 2019)

In terms of its explicit data contents, this update utilized the same data underlying our v5.0 release. However, whereas all prior Pennsylvania releases utilized our original database design, we built Pennsylvania v6.0 on an expanded version of the database design that accompanied our release of Alberta v2.0[7]. Incorporating the Pennsylvania data into this database schema necessitated several enhancements. Notably, the Pennsylvania Department of Environmental Protection provides data on “Well Pads,” basically sites where more than one well may be permitted from the same location. To accommodate these data, we incorporated a new Well Pads table into our database schema. Additionally, the Pennsylvania Department of Environmental Protection reports extensive data regarding inspections, violations, and fines. Comparable data are not available in Alberta, the jurisdiction that served as the basis for our original development of the new database schema. Accordingly, we added new tables related to Inspections, Violations, and Penalties.


We also made several enhancements to our display and reporting of Pennsylvania oil and gas data on WellWiki. First, Pennsylvania has changed its production and waste reporting requirements for unconventional several times, from annual to semiannual to monthly. Additionally, during this transition there was an odd reporting period that spanned from July 2009 to June 2010. Although it is a crude approximation, for reporting purposes on WellWiki’s production table, we divided the values for this period in half, assigning them equally to two new periods we created: 2009-2 and 2010-1. We then aggregated all available data into annual periods, plus reported any data available for years not yet completed. For instance, if well reported production from 2000-0 to 2019-03, meaning for the full calendar year 2000, through to March 2019, we would report annual values for the 2000 to 2018 period, plus monthly values (if any) for the January to March 2019 period. Our Production table now also includes information on the well operator at the time the production was reported. This makes it apparent whether a well has been bought and sold during its operating history.


We also expanded the data displayed on our Operator, and County and Municipality Pages, notably to include “operator” on County and Municipality Pages and "County" and “Municipality” on Operator Pages. This mirrors an enhancement we made as part of our Alberta v3.0[8] update. Finally, this release incorporated data from our Fracking Chemicals v2.0 update. In total, 8,307 Pennsylvania wells reported data through FracFocus; we include those data for each well. Data on Fracking Activities are obtained primarily from FracFocus. For complete details see the Fracking Activities Data Sources[9] page.


Overall, this upgrade resulted in the generation of 270,818 new or modified Pennsylvania Well Pages; 9,002 new or modified Pennsylvania Operator Pages; 1,115 new or modified Pennsylvania County and Municipality Pages; and 3,798 new or modified Pennsylvania Well Pad Pages. The team for this project consisted of Joel Gehman, Cole Miller and Ankesh Kumar. Funding for the project was provided in part by financial support from the Social Sciences and Humanities Research Council of Canada and the University of Alberta, as well as in-kind support University of Alberta School of Business.


References