My team is using Open Foris Arena to collect data for the same site on two devices. Is there a way to merge their data into one form when uploaded to the server? This option would greatly streamline the data collection process. Thank you!

asked 27 Jun, 05:47

Mdeslaur13's gravatar image

Mdeslaur13
2114
accept rate: 0%


12next »

Dear user,
This is a nice feature and we have already started to work on it.
The easiest way to do it is to "warn" the user of the mobile app in case a record with the same key attriutes has been already uploaded to the server by another user, and let him confirm the "merge" of the 2 records.
There are still some issues we are discussing internally:

  • once the records have been "merged", should the mobile app receive the updated/merged version of the record and replace the current one?
  • what will happen to the "common" attributes? For example, suppose that in the same entity, "cluster", you have an attribute "region", should its value be replaced by the last one who has submitted the data? Or should the merging process only insert new entities/attributes that are not already filled in the record on the server?

Please provide your suggestions, any feedback from your side is really appreciated.
Many thanks,
Open Foris Team

permanent link

answered 27 Jun, 09:38

Stefano%20%28OF%29's gravatar image

Stefano (OF) ♦♦
4.8k119
accept rate: 19%

That is great news that this feature will be available. In my situation, two records are created at the same time using the mobile app, then are uploaded to the server at the end of the day. In this case, could the merging process occur on the server once files are uploaded?

My team is working on different entities to avoid the confusion with overlapping information. However, for common attributes that both users may enter (such as site remarks), could it be possible to merge the data to include the entries from both users rather than replacing the information from one user?

permanent link

answered 30 Jun, 00:47

Mdeslaur13's gravatar image

Mdeslaur13
2114
accept rate: 0%

Dear user,
The merging process will occur on the server; however, once it's done, the mobile app will have to download from the server the merged record. In your case, will the users continue to work on the records after they have been merged?
Regarding common attributes, to avoid collision during the import process you could define the site remarks inside a multiple entity (table) with an attribute like "enumerator" as key, so if different enumerators upload the data, the remarks of one operator won't overwrite the other one.
Thanks,
Open Foris Team

permanent link

answered 02 Jul, 09:42

Stefano%20%28OF%29's gravatar image

Stefano (OF) ♦♦
4.8k119
accept rate: 19%

Hello!

I am working on a separate project that would also greatly benefit from the addition of this type of feature. Stefano, here's my input on your questions:

  • Yes, I think the mobile app should recieve the updated/merged version of the record and replace the current one. I think this action should be initiated by the user and not done automatically.
  • I think the merging process should only insert new entities, and surveys should be constructed so that merged data be entered into multiple enities.

It would be great to prevent overwriting when two records with different values for the same single attribute are merged. Perhaps there would be a way to sort through an error like this in the Arena Desktop app before merging? The two records could remain distinct until the survey administrator has had a chance to resolve the conflict within the same attribute between the two records.

The capability to merge records would add enormous benefit for many different projects. Thank you for considering!

Bryant

permanent link

answered 18 Jul, 21:06

bnagelson's gravatar image

bnagelson
2327
accept rate: 0%

I agree with Bryant's suggestions to prevent overwriting of any data, and updating records in the app to reflect the merged records.

Prompting the survey administrator to first preview the merged file before completing the merge, and correct any errors at that time would help prevent any issues with overwriting or sync errors. After this step, having another prompt for the survey administrator to accept the merge and delete the old record(s) could provide an added layer of confirmation in the process.

It could also be helpful to have an option to recover previously merged files in the event something was overlooked and the original record needed to be returned to. Or a prompt to check if the administrator would like to export or back-up the original data at the time of the merge.

permanent link

answered 20 Jul, 01:06

Mdeslaur13's gravatar image

Mdeslaur13
2114
accept rate: 0%

Dear users,
From our point of view it would be much easier if the merging could be done in the most automatized way possible.
As Bryant was suggesting, the easiest thing to do is to design the survey in a way that is "merge ready", so for example:

  • if you plan to collect trees inside the same plot with different devices, you can think about dividing the plot into subplots and assigning each subplot to different devices;
  • if you want to "join" information like site remarks, you should define them into a multiple entity, with the enumerator as key attribute;
  • attributes already filled using one device can be left blank on the other one, and during the merge process they can stay as they are, otherwise they can be overwritten if the 2nd device has different information;

The easiest thing to do at the moment could be to allow merging only server side (by a user with survey administrator role for example), giving the possibility to select the records to merge from the records list and previewing the result before confirming it; once merged, one of the 2 records could be marked as "merged" and the same record in the device can be deleted and replaced with the merged version of it.
Does it sound good to you? Please let us know.
Many thanks,
Open Foris Team

permanent link

answered 24 Jul, 09:30

Stefano%20%28OF%29's gravatar image

Stefano (OF) ♦♦
4.8k119
accept rate: 19%

I like the idea of merging records from a multiple entity. It would be more useful to us if we did not need to assign trees into subplots within the survey before entering data. Ideally, each tree could be entered into the same multiple entity on different devices and merged afterwards. I would assume that each tree would still need to be uniquely identifiable by a set of user-defined key attributes in order to avoid conflicts.

There's still the question of how to handle a single attribute with entries from two different devices. I like Michelle's idea of allowing the survey administrator to preview and control the merging process where conflicts exist.

permanent link

answered 31 Jul, 21:10

bnagelson's gravatar image

bnagelson
2327
accept rate: 0%

Dear Bryant,
As you are saying, the important thing is to uniquely identify the entities, so if you define them inside a subplot entity or if you define another key attribute, it will be the same.
The easiest way to handle this at the moment is to allow merging records only from the server/desktop, being able to preview the result of the merge and confirm it eventually (without having the possibility to control the merging process, because it would be to complex to do it for the user, expecially when the records are big).
Regarding the single attributes in the same entity, the way we will proceed is to replace the existing value in case it has been updated in the newer record, or if its value was empty.
Once merge is confirmed, the user can be asked to delete the old record.
Let us know if it would fit your needs.
Many thanks,
Open Foris Team

permanent link

answered 02 Aug, 15:03

Stefano%20%28OF%29's gravatar image

Stefano (OF) ♦♦
4.8k119
accept rate: 19%

Hi Stefano,

The configuration you describe here sounds like it will meet my needs. But just to be sure, I'd like to explain my project and how I plan to tailor our survey to utilize the merging functionality.

The project involves collecting data on planted trees in 0.5 hectare plots. Trees are planted in rows within plots, with each plot containing 15-20 rows and each row containing 80-150 trees. It is necessary to collect data in different rows simultaneously to maximize effeciency. In a typical day, we will use two or three different devices to collect all the data in a single plot. At the smallest level, a single plot could generate as many as 3,000 tree records. Each tree is uniquely identified within rows and each row is uniquely identified within plots.

Here is how I imagine the survey to be structured: Each tree is a single record within a multiple entity for the row, and each row is a record of a multiple entity for the plot. When merged, a single record will exist for the plot that contains all the sub-records for each row and all sub-sub-records for each tree. There will be single attributes at the plot-level, but we would communicate between our data collectors so that only one device is used to enter these single attributes (aside from the key attributes that uniquely identify the plot and are used for merging).

Thank you for so carefully considering this functionality. It will greatly enhance our workflow!

Bryant

permanent link

answered 02 Aug, 16:04

bnagelson's gravatar image

bnagelson
2327
accept rate: 0%

Stefano,

Thank you for creating the merging feature on the Experiments app; it greatly streamlined our workflow for data collection and management. I have been careful to ensure there is no overlapping data that would overwrite, so I've had no issues there so far.

Prior to this feature merging our records, each plot had two records (each with missing fields of data) that I was planning on merging manually through the server. I have tried to start this process, but it seems like it will take some time and is not as intuitive as I thought it to be. Is it possible to have an option to merge records through the server as well as the Experiments app? This would greatly enhance my workflow while cleaning data, specifically for the records that my team created prior to the merging feature becoming available in the Experiments app.

Thank you for your continued work on this feature!

Michelle

permanent link

answered 23 Aug, 02:55

Mdeslaur13's gravatar image

Mdeslaur13
2114
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×58
×46

question asked: 27 Jun, 05:47

question was seen: 1,402 times

last updated: 30 Aug, 14:51