Category Archives: Tutorials

LiDAR Training

Source: U.S. Forest Service Remote Sensing Applications Center

Lidar Training

Lidar Supporting Documentation

Setting up Anaconda, PySAL with ArcGIS Python environment

This tutorial shows you how to set up conda environment to work with ArcGIS 10.4 and ArcGIS Pro 1.3. You can download a pdf copy of the tutorial with screenshots HERE. At the 2016 Esri International User Conference in San Diego last month, Esri released ArcGIS Pro 1.3, which can now use conda for packaging Python libraries. This allows support of python under multiple Python environments. You no longer need to install a separate Python install to get the full Python capability with ArcGIS as you did with past versions.

Workflow to set up Anaconda with ArcGIS 10.4

  • Install Anaconda without fouling the Windows environment (paths, registry) to break Esri’s python stack
  • Configure Anaconda with the particular add-ons you want, and
  • Configure ArcGIS’s Python so that it is aware of the appropriate Anaconda content.

1) Install Anaconda for All Users

  1. Go to http://continuum.io/downloads
  2. Download the 32-bit version of Anaconda (Python 2.7)
  3. In the install dialogs:
    • Select install for All Users
    • conda01
    • Install to a folder by default (C:\Anaconda2)
    • conda02
    • IMPORTANT: To avoid breaking ArcGIS (or other software), uncheck the checkboxes (a) make Anaconda the default Python and (b) add Anaconda’s Python to the PATH.
    • conda03
  4. Go to Start > All Programs(apps) > Anaconda2(32-bit) > Anaconda Prompt. Right click, run as administrator

2) Configure an Anaconda environment for use with ArcGIS

  1. Find the versions of numpy and matplotlib ArcGIS is using.

Open ArcMap and its Python window, and enter these commands:

    • >>> import sys, numpy, matplotlib
    • >>> print(sys.version, numpy.__version__, matplotlib.__version__)
    • (‘2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]’, ‘1.9.2’, ‘1.4.3’)
    • conda04
  1. Create an Anaconda environment that is compatible with ArcGIS
    • Get to the Anaconda Command Prompt (Start > All Programs(apps) > Anaconda2(32-bit), pick “Anaconda Prompt“), Right click, Run as Administrator.
    • Type (depending on ArcGIS version, I am using ArcGIS 10.4 as an example here):
    • “conda create -n arc104 python=2.7.10 numpy=1.9.2 matplotlib=1.4.3 pyparsing xlrd xlwt pandas scipy ipython ipython-notebook ipython-qtconsole”
    • conda05
    • Enter y to proceed.
    • Anaconda’s conda command will then set up an environment subdirectory, ex: ” C:\Anaconda2\envs\arc104 “, installing the downloaded packages into it.
    • conda06
  2. Test the virtual environment
    • At the Anaconda Command Prompt, type: activate arc104
    • Type: conda list. You can see the list of packages installed.conda07
  3. Add more packages
    • You can add more packages using conda install, but make sure you specify version numbers for these that won’t change the environment’s version of python or numpy (or ArcGIS will not be able to use that environment anymore).
    • Let’s add the Python Spatial Analysis Library (pysal) module.
    • Type the following command at the Anaconda Prompt:”conda install -n arc104 python=2.7.10 numpy=1.9.2 matplotlib=1.4.3 pysal”
    • conda08

3) Configure ArcGIS to see Anaconda and vice versa

  1. Anaconda Python to ArcPy
    • Copy the Desktop10.4.pth file to the Anaconda environment site-packages folder:
    • From: C:\Python27\ArcGIS10.4\Lib\site-packages\Desktop10.4.pth
    • To: C:\Anaconda2\envs\arc104\Lib\site-packages\Desktop10.4.pth
  2. Arcpy to Anaconda Python
    • Create a zconda.pth (path) file with the content “C:\Anaconda\envs\arc104\lib\site-packages” in it.
    • Then copy zconda.pth to C:\Python27\ArcGIS10.4\Lib\site-packages
  3. Testing in ArcMap
    • As a regular user, start ArcMap, open the Python window
    • type “import pysal”
    • type “pysal.” A popup menu with a list of pysal-provided functions is a pretty good sign the installation succeeded.
    • conda09
  4. Testing in PyCharm
    • Start PyCharm, in File\Settings…, choose Project then Project Interpreter
    • Ignore the drop down list for Project Interpreter, and click the cog button to Add Local, and in the file browser pick C:\Anaconda2\envs\arc104\python.exe
    • conda10
    • To run your script, right click it in the Project window, and choose either Run or Debug
    • Restart PyCharm for the Python Console to use the arc104 environment.
    • conda11

Workflow to set up Anaconda with ArcGIS Pro 1.3

1) Create an Anaconda environment that is compatible with ArcGIS Pro

  • Copy the folder arcgispro-py3 from C:\Program Files\ArcGIS\Pro\bin\Python\envs and paste to C:\Anaconda2\envs
  • Rename the copied folder arcgispro-py3 in C:\Anaconda2\envs to arcpro

2) Test the virtual environment

  • At the Anaconda Command Prompt, type: activate arcpro
  • Type: conda list. You can see the list of packages installed

3) Add more packages

  • Let’s add the Python Spatial Analysis Library (pysal) module.
  • Type the following command at the Anaconda Prompt:”conda install pysal”

4) Configure ArcGIS to see Anaconda and vice versa

  • Arcpy to Anaconda Python
    • Create a zconda.pth (path) file with the content “C:\Anaconda2\envs\arcpro\lib\site-packages” in it.
    • Then Copy zconda.pth to C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages
  • Testing in ArcGIS Pro
    • Start ArcGIS Pro, open the Python window
    • type “import pysal”
    • type “pysal.” A popup menu with a list of pysal-provided functions is a pretty good sign the install succeeded.
    • conda12
    • conda13
  •  Testing in PyCharm
    • Start PyCharm, in File\Settings…, choose Project then Project Interpreter
    • Ignore the drop down list for Project Interpreter, and click the cog button to Add Local, and in the file browser pick C:\Anaconda2\envs\arcpro\python.exe
    • conda14
    • To run your script, right click it in the Project window, and choose either Run or Debug
    • Restart PyCharm for the Python Console to use the arcpro environment.

References

  1. USGS: https://goo.gl/xd6xz2
  2. Esri: https://goo.gl/tYGHrw
  3. GeoNet: https://goo.gl/mTLWMG
  4. UC-Davis: http://goo.gl/3bdbwz

R script for updating student grades on Blackboard

This might be of interest to some of you teaching large enrollment courses and using scantrons for quizzes/exams. I developed a script using R programming language to automatically extract scores from ITS test scoring results and upload the grades to Blackboard.

The script needs two CSV format input files: the student info file from Blackboard (Full Grade Center -> Work Offline – Download) and the ITS test scoring results (convert the Excel file to CSV). It takes less than one second to get the results.

Feel free to let me know if you have any questions.

########################################

BBfile <- file.choose()  #”roster.csv”    ### The file downloaded from Blackboard
ITSfile <- file.choose()   #”result.csv”   ### The file received from ITS scantron results

# BBfile <- “roster.csv”
# ITSfile <- “result.csv”
output <- “score.csv”
scale.factor <- 1  ### scale factor multiplied by the scantron results.
### Extract students’ fullname from Blackboard roster
roster <- read.csv(BBfile,header = TRUE,stringsAsFactors = FALSE)
str(roster)
roster$firstname = as.character(lapply(strsplit(as.character(roster$First.Name), split=” “), “[“, 1))
roster$fullname <- tolower(paste(roster$Last.Name,roster$firstname,sep=””))
### read the ITS results
df <- read.csv(ITSfile,stringsAsFactors = FALSE)
df <- df[nchar(gsub(” “,””,df$X))>0,]
df <- df[!is.na(as.numeric(df$X.5)),c(“X”,”X.2″)]
colnames(df) <- c(“Name”,”Score”)
df$Score <- as.numeric(df$Score) * scale.factor
str(df)
### extract student names from ITS results

lastname <- as.character(lapply(strsplit(as.character(df$Name), split=” “), “[“, 1))
firstname <- as.character(lapply(strsplit(as.character(df$Name), split=” “), “[“, 2))
df$fullname  <- tolower(paste(lastname,firstname,sep = “”))
### match student names from Blackboard and ITS
m.x <- merge(roster,df,by = “fullname”,all.x = TRUE)
fix(m.x)
m.x$raw <- m.x$Score / scale.factor
### save the results to csv file
write.csv(m.x,output,na = “”,row.names = FALSE)
m.y <- merge(roster,df,by = “fullname”,all.x = TRUE,all.y = TRUE)
m.y.sub <- m.y[is.na(m.y$Last.Name), ]
score <- read.csv(output,header = TRUE,stringsAsFactors = FALSE)
summary(score)

########################################

15 Free Satellite Imagery Data Sources

Source: http://gisgeography.com/free-satellite-imagery-data-list/

If you want free satellite data, there’s no better way to do it then to follow this incredibly useful guide. Ranked from top to lower tier, here are your go-to free satellite imagery sources. Take a look at our list of eyes from the sky.

1 USGS Earth Explorer – Unlock the Power of Landsat and More

USGS Earth Explorer

Whether you live in the United States, in the Arctic circle or an obscure country like Transnistria, we can all appreciate the abundance of data theUSGS Earth Explorer has to offer.

We’ve relentlessly hyped USGS Earth Explorerhere, here and here… .and we’re about to do it again…

From no data to hyperspectral data, USGS is the undisputed world champion of free satellite data providers. Here’s why:

  • Access to Landsat satellite data – a legacy that goes unmatched. 40-years of history of our Earth with consistent spectral bands.
  • Vertically position yourself with NASA’s ASTER and Shuttle Radar Topography Missions global Digital Elevation Models.
  • Gain full access to NASA’s Land Data Products and Services including Hyperion’s hyperspectral data, MODIS & AVHRR land surface reflectance and disperse Radar data.

We sound like a broken record. But USGS Earth Explorer is a world class source of free satellite data. Regardless where you live, you NEED to look at the USGS Earth Explorer.

Read More: How to Download Free Landsat Imagery from the USGS Earth Explorer:

2 ESA’s Sentinel Mission – New Leader in Free High Resolution Data?

Read the rest of this entry

Tutorials for processing Sentinel-1data

I just started exploring Sentinel-1 SAR data for my research on wetlands and water resources. Here are some useful resources I found:

About the Sentinel-1 mission:

Blog posts:

Advanced training course on the use of Sentinel-1 SAR data:

Sentinel-1 data analysis using PCI Geomatica:

Synthetic Aperture Radar: Of Bats and Flying Pianos:

  • An amusing introduction to radar remote sensing from satellites, with the concept of “range Doppler” image formation described using entertaining audio-video animations.


How to Download Sentinel Satellite Data for Free

For those who are interested in using the Sentinel-1 and Sentinel-2 Satellite Data from the European Space Agency’s Copernicus Programme, please check out this blog at http://gisgeography.com/how-to-download-sentinel-satellite-data/. Note that Sentinel 2A multispectral data has a 10-m spatial resolution, which is much better than the Landsat 8 with 30-m resolution.

How-To-Download-Sentinel-Satellite-Data

ProductDownload-Sentinel_SciDH

What are the Spectral Bands of Sentinel 2A and 2B?

The spectral and spatial resolution of Sentinel 2A are listed below. There are 13 bands in total. Four spectral bands have a 10 meter resolution. Six bands have a 20 meter resolution. And the remaining 3 have a spatial resolution of 60 meters.

Here are the spectral band details for Sentinel 2A:

Source: SENTINEL-2 Spatial Resolution

Each single satellite revisit time is 10 days. Because there are two satellites (Sentinel 2A and 2B), this means it has a combined constellation revisit of 5 days.

Access Landsat Imagery in ArcGIS Pro

Related post: Landsat 8 data on Amazon AWS

Here’s how you can access the Landsat archive which is hosted on Amazon through ArcGIS Pro.

  1. Open the Projects Pane
  2. Click on Portal and then on the cloud icon on the far right to see everything that’s on your portal.
  3. Search for “Multispectral Landsat” in the search bar.
  4. Drag and drop.

To take advantage of the archive, we’ve designed a number of pre-defined band combinations and indices. You can find these from the Data tab, under Processing Templates.

Play around with these. Here’s some cool stuff that I found poking around in Alaska for a few minutes.

Read the rest of this entry

ArcGIS Terrain Toolbox

Terrain Tools Sample v1.0 beta:  A suite of Geoprocessing tools to produce cartographic effects for terrain representation

Download it here.

Source: http://blogs.esri.com/esri/arcgis/2015/05/21/take-your-terrain-mapping-to-new-heights/

Take your terrain mapping to new heights

by Kenneth Field, Senior Cartographic Product Engineer

Standard techniques for representing terrain, like a hillshade, are adequate for many applications, but you may want to represent terrain under different lighting conditions, or perhaps use more artistic techniques. In these cases you might need to go a little further than creating a simple hillshade. For this reason (and because we like making tools that extend what we can do!) we’ve built a new toolbox, called Terrain Tools, that helps you take your terrain mapping to new heights.

Cartographic research often develops techniques, models and tools that supplement or extend what you find in software out-of-the-box. They are often difficult to find, hidden in journal articles or cumbersome to implement. We’ve brought together some of these ideas and workflows in the Terrain Tools toolbox that provides capabilities for creating alternative terrain representations in ArcGIS (ArcMap and ArcGIS Pro). Terrain Tools are designed to extend your out-of-the-box toolkit for representing terrain in GIS by encouraging you to be more creative; how to think more critically about design choices; and how to go beyond the defaults.

You can download a zip file from ArcGIS Online that contains the toolbox, sample data, documentation and also an ArcMap Map Document and ArcGIS Pro Project. The Map Document and Pro Project also include results layers so you can see how the tools work before you use them on your own data. It’s important to note that the sample results show the results of running the tools using the default output. They are a starting point and many of the tools give you the flexibility to modify parameters and customize your own output. The documentation is in workshop format that includes discussion of each tool and instructions for use. Because the tools are written in Python they can be viewed, modified and used as a starting point for further development.

Here’s a brief look at some of the output you’ll get from running Terrain Tools.

Terrain Tools incorporates a few previously available tools which were available as models (built originally using ModelBuilder). They have been rewritten and optimized as Python scripts which improves stability and speed of processing. Additionally, a range of new tools have been either written (from the original published algorithms) or optimized as Python scripts from code samples.

One of the highlights of Terrain Tools is the new Cluster Hillshade which provides you with the ability to make spectacularly detailed and artistic hillshades with your own data. This is just about as close as you can get with an automated process to classic hand-drawn hillshading – and all from just a Digital Elevation Model input and a click of a mouse.

The Tanaka method for creating Illuminated Contours and Filled Contours are also included, here depicted one on top of the other:

Thematic maps haven’t been ignored. It’s perfectly possible to run the tools for any input raster (e.g. a statistical surface rather than a DEM) but there’s also a specific 3D Choropleth tool, useful for adding depth to a choropleth, encoding a second piece of information or as a way to see variation within a single class interval:

There’s plenty more tools to explore and, of course, the scripts are entirely open to being customised further. Here’s the full list of what’s included in the Terrain Tools sample:
Read the rest of this entry

ENVI Unsupervised Classification Workflow

Software used: ENVI 5.1

The ISODATA method for unsupervised classification starts by calculating class means evenly distributed in the data space, then iteratively clusters the remaining pixels using minimum distance techniques. Each iteration recalculates means and reclassifies pixels with respect to the new means. This process continues until the percentage of pixels that change classes during an iteration is less than the change threshold or the maximum number of iterations is reached.

  1. Start ENVI.
  2. From the Toolbox, select Classification > Classification Workflow. The File Selection panel appears.

     ISO0

  1. Click Browse. The File Selection dialog appears.

ISO01

Read the rest of this entry

Surface Temperature from Landsat Data: a New Lab Using the Semi-Automatic Classification Plugin

A very detailed tutorial for deriving surface temperature from Landsat data using QGIS and Semi-Automatic Classification Plugin.

Source: From GIS to Remote Sensing: Surface Temperature from Landsat Data: a New Lab Using the Semi-Automatic Classification Plugin.

Download the tutorial pdf (link)

In a previous post I have illustrated how to estimate land surface temperature using Landsat images and the Semi-Automatic Classification Plugin.
I was very pleased when Katie Fankhauser, a graduate student at Portland State University, informed me that she was preparing a lab, inspired by that tutorial, about how to determine ground surface temperature using satellite imagery and my plugin.
The document that she prepared provides background information about remote sensing and Landsat imagery (such as conversion of Landsat images to TOA reflectance and brightness temperature), and describes all the required phases about:
– download of software and data;
– data processing and supervised classification of land cover;
– calculation of an emissivity raster and estimation surface temperature.
Temperature calculated for a Landsat image of Portland (data available from the U.S. Geological Survey)


The image used in the lab is a Landsat image of Portland (OR, USA). The processing of data is described step by step, from the ROI creation to the raster calculation, which is ideal also for students that have little experience with remote sensing.
The lab document (pdf file) is freely available at the following link:
http://web.cecs.pdx.edu/~derekt/Classes/ME%20411%20Winter%202015/Laboratory/ME411_RemoteSensingLab4.pdf
This lab was prepared by Katie Fankhauser, and Evan Thomas who is Assistant Professor of the course Mechanical Engineering Measurements at the Portland State University, Sustainable Water, Energy and Environmental Technologies Laboratory. Very kindly, they credited also me as an author of this lab and allowed me to share the document.
Katie is currently working for a health campaign that aims to reduce the amount of wood fuel use consumed by traditional stone fires, and she is involved in ground truthing satellite-derived land surface temperature to study the rates of deforestation in Rwanda.
Evan Thomas holds a Ph.D. in Aerospace Engineering Sciences, and in particular he is the Director of the Sweet (Sustainable Water, Energy and Environmental Technologies) Laboratory, a very interesting and worthy project (www.sweetlab.org):
“At Portland State, the SweetLab designs and tests sustainable life support technologies for spacecraft and developing countries. The SweetLab’s current primary focus is developing and implementing remotely accessible instrumented monitoring technologies designed to improve the collection of effectiveness evidence in global health programs, including high efficiency cookstoves, water pumps, household water filters, sanitation systems, pedestrian footbridges and other developing world appropriate technologies. The SweetLab has projects in India, Nepal, Indonesia, the Philippines, Rwanda, Kenya, Uganda, Haiti and other countries with partners including the Gates Foundation, USAID, Mercy Corps, the Lemelson Foundation, the Global Alliance for Clean Cookstoves, and DelAgua. The SweetLab also has on-going work with the NASA-Johnson Space Center on microgravity fluid management systems”.
I am really glad and honored that my work can be useful for courses about environmental sustainability like this. Also, I hope that the upcoming new version of the plugin will allow for improved environmental analyses.
I would like to thank very much Katie and Evan and I hope that there will be other opportunities of cooperation in the future.

If one is interested in sharing the work done using the Semi-Automatic Classification Plugin, please contact me at the Facebook group or the Google+ Community.
REDD+ for the Guiana Shield

Technical Cooperation Project

LidarBlog.com

Dr. Qiusheng Wu @ SUNY Binghamton

Writing Science

How to write papers that get cited and proposals that get funded

GIS In Ecology

Providing Training, Advice And Consultation On The Use Of GIS In Ecology

metrhispanic

On cities, land, ...

GeoAcademy

Open GIS: No Bounds

Scientia Plus Conscientia

Thoughts on Science and Nature

r4hydrology

Learning hydrology with R

Karl Hennermann

GIS at the University of Manchester

GIS and Science

Applications of geospatial technology for scientific research and understanding.

GISblog.com

GIS, Mapping, Remote Sensing, Geodata, Geospatial news

Whitebox Geospatial Analysis Tools

Open-source GIS development and spatial analysis with Whitebox GAT

TopoToolbox

MATLAB-based software for topographic analysis

Anything Geospatial – AnyGeo

Dr. Qiusheng Wu @ SUNY Binghamton

GeospatialPython.com

Dr. Qiusheng Wu @ SUNY Binghamton

Another GIS Blog

Dr. Qiusheng Wu @ SUNY Binghamton

ArcPy Café

Get all your ArcGIS Python Recipes here!

Planet Python

Dr. Qiusheng Wu @ SUNY Binghamton