Category Archives: Algorithms
My new peer-reviewed article titled “Automated delineation of karst sinkholes from LiDAR-derived digital elevation models” has been published in the latest issue of Geomorphology. You can download a free online copy using this link: http://authors.elsevier.com/a/1T3d9_,Oh6mAl8 (expires on July 8, 2016). In this paper, we present a localized contour tree method for automated extraction of sinkholes in karst landscapes. The study area was Fillmore County in southeastern Minnesota, USA. See some figures below:
This might be of interest to some of you teaching large enrollment courses and using scantrons for quizzes/exams. I developed a script using R programming language to automatically extract scores from ITS test scoring results and upload the grades to Blackboard.
The script needs two CSV format input files: the student info file from Blackboard (Full Grade Center -> Work Offline – Download) and the ITS test scoring results (convert the Excel file to CSV). It takes less than one second to get the results.
Feel free to let me know if you have any questions.
BBfile <- file.choose() #”roster.csv” ### The file downloaded from Blackboard
ITSfile <- file.choose() #”result.csv” ### The file received from ITS scantron results
# BBfile <- “roster.csv”
# ITSfile <- “result.csv”
output <- “score.csv”
scale.factor <- 1 ### scale factor multiplied by the scantron results.
### Extract students’ fullname from Blackboard roster
roster <- read.csv(BBfile,header = TRUE,stringsAsFactors = FALSE)
roster$firstname = as.character(lapply(strsplit(as.character(roster$First.Name), split=” “), “[“, 1))
roster$fullname <- tolower(paste(roster$Last.Name,roster$firstname,sep=””))
### read the ITS results
df <- read.csv(ITSfile,stringsAsFactors = FALSE)
df <- df[nchar(gsub(” “,””,df$X))>0,]
df <- df[!is.na(as.numeric(df$X.5)),c(“X”,”X.2″)]
colnames(df) <- c(“Name”,”Score”)
df$Score <- as.numeric(df$Score) * scale.factor
### extract student names from ITS results
lastname <- as.character(lapply(strsplit(as.character(df$Name), split=” “), “[“, 1))
firstname <- as.character(lapply(strsplit(as.character(df$Name), split=” “), “[“, 2))
df$fullname <- tolower(paste(lastname,firstname,sep = “”))
### match student names from Blackboard and ITS
m.x <- merge(roster,df,by = “fullname”,all.x = TRUE)
m.x$raw <- m.x$Score / scale.factor
### save the results to csv file
write.csv(m.x,output,na = “”,row.names = FALSE)
m.y <- merge(roster,df,by = “fullname”,all.x = TRUE,all.y = TRUE)
m.y.sub <- m.y[is.na(m.y$Last.Name), ]
score <- read.csv(output,header = TRUE,stringsAsFactors = FALSE)
I just came across an interesting article: Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance. – An open-source exploration of the city’s neighborhoods, nightlife, airport traffic, and more, through the lens of publicly available taxi and Uber data.
Quoted from the author Todd W. Schneider :
“The New York City Taxi & Limousine Commission has released a staggeringly detailed historical dataset covering over 1.1 billion individual taxi trips in the city from January 2009 through June 2015. Taken as a whole, the detailed trip-level data is more than just a vast list of taxi pickup and drop off coordinates: it’s a story of New York. How bad is the rush hour traffic from Midtown to JFK? Where does the Bridge and Tunnel crowd hang out on Saturday nights? What time do investment bankers get to work? How has Uber changed the landscape for taxis? And could Bruce Willis and Samuel L. Jackson have made it from 72nd and Broadway to Wall Street in less than 30 minutes? The dataset addresses all of these questions and many more.
I mapped the coordinates of every trip to local census tracts and neighborhoods, then set about in an attempt to extract stories and meaning from the data. This post covers a lot, but for those who want to pursue more analysis on their own: everything in this post—the data, software, and code—is freely available. Full instructions to download and analyze the data for yourself are available on GitHub.”
My new peer-reviewed article titled “A localized contour tree method for deriving geometric and topological properties of complex surface depressions based on high-resolution topographical data” has been published in the latest issue of International Journal of Geographical Information Science, Issue 12. You can download a free online copy using this link: http://www.tandfonline.com/eprint/r34t6WVptm2hUgfjQc5r/full. In this paper, we developed a localized contour tree method that is able to fully exploit high-resolution topographical data for detecting, delineating and characterizing surface depressions across scales with a multitude of geometric and topological properties. See some figures below:
“GoSpatial is a command-line interface program for analyzing and manipulating geospatial data. It has been developed using the Go programming language and is compiled to native code. The project is experimental and is intended to provide additional analytical support for the Whitebox Geospatial Analysis Tools open-source GIS software. GoSpatial can however be run completely independent of any other software and is run from a single self-contained executable file.”
There are a few exciting announcements related to new developments on the Whitebox GAT front. The first is that due to changes in the Google Code practices (it has become ‘read only’ and is no longer accepting new code commits), the Whitebox project has moved its source code repository to GitHub. I think that this will eventually make for improved source code management, although there may be some initial transition issues that we’ll need to work past. Some of the documentation will have to be updated to reflect this change.
The second announcement, which I am most excited about, is that I have finally found the time to update the Whitebox GAT website. There is a fresh new and more professional look to the site. I hope you enjoy the new webpage and as always, feedback is welcome. What would you like to see added or changed?
View original post 88 more words
R-bridge for ArcGIS: https://github.com/R-ArcGIS/r-bridge
Today at the Esri User Conference in San Diego, Esri announced a new initiative to build a collaborative community for R and ArcGIS users.
Esri has been teaching and promoting integration with R at the User Conference and Developer Summit for several years. During this time we have seen significant increase in interest, and received useful feedback from our ArcGIS users and R users about a variety of needs and techniques for integrating ArcGIS and R. Based upon this feedback, we are working with ArcGIS and R users to develop a community to promote learning, sharing, and collaboration. This community will include a repository of free, open source, R scripts, geoprocessing tools, and tutorials.
I recently sat down with Steve Kopp, Senior Product Engineer on the spatial analysis team, and Dawn Wright, Esri’s Chief Scientist to talk about what this focus on building a bridge to the R community means for ArcGIS users and other users of R.
Matt Artz: What is R?
Steve Kopp: R (aka the R Project for Statistical Computing) is an extremely popular and the fastest growing environment for statistical computing. In addition to the core R software, it includes a more than 6,000 community-contributed packages for solving a wide range of statistical problems, including a variety of spatial statistical data analysis methods.
Dawn Wright: R is widely used by environmental scientists of all stripes, as well as statisticians. Since R has limited data management and mapping capabilities, many of our users find good synergy in using R and ArcGIS together.
Matt Artz: Does the ArcGIS community use R today?
Steve Kopp: Yes, R has become very popular in the ArcGIS community over the last several years. Many in our user community have been asking for a mix of its functionality with our own, as well as better code-sharing interaction with the R community.
Dawn Wright: A great example from the marine ecology community is Duke University’s Marine Geospatial Ecology Tools, where they have already long since moved forward with integrating R and ArcGIS for Desktop for some time.
Matt Artz: What is the R – ArcGIS Bridge?
Steve Kopp: This is a free, open source R package which allows ArcGIS and R to dynamically access data without creating intermediate files on disk.
Matt Artz: Why did Esri build the R – ArcGIS Bridge?
Steve Kopp: It was built for three reasons: to improve the performance and scalability of projects which combine R and ArcGIS; to create a developer experience that was simple and familiar to the R user; and to enable an end-user experience that is familiar to the ArcGIS user.
Dawn Wright: The bottom line is that this project is about helping our user community and the R user community to be more successful combining these technologies.
Matt Artz: So is this initiative just some software code?
Steve Kopp: No, the R – ArcGIS Bridge is simply some enabling technology, the real effort and value of the R – ArcGIS Community initiative will be in the development and sharing of useful tools, tutorials, and tips. It’s a free, open source library which makes it fast and easy to move data between ArcGIS and R, and additional work which makes it possible to run an R script from an ArcGIS geoprocessing tool.
Dawn Wright: This community will be important and useful for R users who need to access ArcGIS data, for ArcGIS users who need to access R analysis capabilities from ArcGIS, and for developers who are familiar with both ArcGIS and R who want to build integrated tools or applications to share with the community.
Steve Kopp: The community of tools will be user developed and user driven. Esri will develop a few sample toolboxes and tutorials, but our primary interest is to facilitate the community and help them build what they find useful.
Matt Artz: How do you see the ArcGIS community using the R – ArcGIS Bridge? What does it give them that they don’t have today?
Steve Kopp: The R – ArcGIS Bridge allows developers with experience with R and ArcGIS to create custom tools and toolboxes that integrate ArcGIS and R, both for their own use, and for building toolboxes to share with others both within their organization and with other ArcGIS users.
Dawn Wright: R developers can quickly access ArcGIS datasets from within R, save R results back to ArcGIS datasets and tables, and easily convert between ArcGIS datasets and their equivalent representations in R.
Steve Kopp: It allows our users to integrate R into their workflows, without necessarily learning the R programming language directly.
Matt Artz: What about the R user who doesn’t use ArcGIS?
Steve Kopp: It’s not uncommon in an organization for a non-GIS person to need to be able to work with GIS data; for these people, they will be able to use the bridge to directly access ArcGIS data without creating intermediate shapefiles or tables, and without needing to know any ArcGIS.
Matt Artz: How can people start using the R Bridge?
Steve Kopp: The R – ArcGIS community samples, tutorials, and bridge are all part of a public GitHub community site similar to other Esri open source projects. And if you happen to be at the Esri User Conference in San Diego this week, this project will be discussed as part of a workshop on Wednesday.
I am pleased to announce that our paper titled “A localized contour tree method for deriving geometric and topological properties of complex surface depressions based on high-resolution topographical data” has been published online in the International Journal of Geographical Information Science.
In addition, I am also happy to announce that I have successfully defended my Ph.D. dissertation and I will be joining the Department of Geography at State University of New York at Binghamton (SUNY-Binghamton) as a tenure-track assistant professor beginning August 2015.
An interesting article on geotagged tweets with open-source tools!
My co-authored article on wetland classification just got published online: http://www.mdpi.com/2072-4292/6/12/12187