Inferring Patterns from Uncensored Data
Preparing Our Data
Our study involved data from pregnant women enrolled into a number of trials evaluating interventions to combat malaria in pregnancy. This comprised the following variables at first ANC visit: HB concentrations, infection status by PCR, gestational age and gravidity. We also use ANC prevalence in primigravidae as a marker to determine our model-based estimates of patterns of prior exposure during pregnancy. The function generate_country_data_list_censor creates a list that involves both the data (here the synthetic data from data_generation.Rmd) and pre-calculated aggregated number of PCR samples and PCR positives in primigravidae within the dataset (to reduce computational time)
# Generate the uncensored data list
uncensored_data_list <- generate_country_data_list_noncensor(data_df = synthetic_data_df$data_df)
# Display the first few rows of the data
knitr::kable(head(uncensored_data_list$country_df))| gravidity | gestage | adj_gestage | malaria | hb_level |
|---|---|---|---|---|
| 1 | 120 | 41 | 0 | 14.844876 |
| 1 | 179 | 100 | 1 | 6.822151 |
| 1 | 139 | 60 | 1 | 8.534953 |
| 1 | 170 | 91 | 0 | 8.939274 |
| 1 | 176 | 97 | 1 | 6.326073 |
| 1 | 88 | 9 | 0 | 13.769398 |
## and the additional prevalence data added to the list
knitr::kable((uncensored_data_list$prev_data))| positive | total |
|---|---|
| 2805 | 5501 |
Defining Parameters
Our approach uses the drjacoby package which requires specific templates for parameters. Here we use the function ‘generate_df_params_non_censor’ to set up a template of parameters required for this package. These parameters include:
- Natural dynamics by gestational age (y_knots) and by gravidity (Gx_non_infect)
- Dynamics of malaria impact by gestational age (mal_knots) and by immune status (shape/scale_hill)
- Residual individual-level variability (HB_sigma)
- A parameter for malaria prevalence at ANC in primigravidae to capture uncertainty in our sample.
# Generate parameter file for uncensored data
df_params_non_censor <- generate_df_params_non_censor()
# Display the parameter file
knitr::kable(df_params_non_censor)| name | min | max | |
|---|---|---|---|
| 1 | HB_sigma | 0.100 | 10.000 |
| 2 | shape_hill | 0.010 | 10.000 |
| 3 | scale_hill | 0.010 | 10.000 |
| name1 | G2_non_infect | -5.000 | 5.000 |
| name2 | G3_non_infect | -5.000 | 5.000 |
| name3 | G4_non_infect | -5.000 | 5.000 |
| name4 | G5_non_infect | -5.000 | 5.000 |
| name5 | G6_non_infect | -5.000 | 5.000 |
| name11 | y_knot_1 | 5.000 | 15.000 |
| name21 | y_knot_2 | 5.000 | 15.000 |
| name31 | y_knot_3 | 5.000 | 15.000 |
| name12 | mal_knot_1 | -10.000 | 10.000 |
| name22 | mal_knot_2 | -10.000 | 10.000 |
| name32 | mal_knot_3 | -10.000 | 10.000 |
| name | log_odds_malaria_prevalence_G1 | -4.595 | 1.736 |
Running the Model on Uncensored Data
With the data prepared and parameters defined, we now run our model on the uncensored dataset. This will help us validate our model’s ability to capture the true underlying patterns without any biases introduced by data censorship.
run_noncensored_data <- run_mcmc(
data = uncensored_data_list,
df_params = df_params_non_censor,
misc = list(
gestage_min = 80,
gestage_max = 200,
cutoff = NULL,
inf_history = selected_inf_history
),
loglike = r_loglike,
logprior = r_logprior_noncensor,
burnin = 1000,
samples = 10000,
chains = 1
)
#> MCMC chain 1
#> burn-in
#> | |= | 1% | |= | 2% | |== | 3% | |=== | 4% | |==== | 5% | |==== | 6% | |===== | 7% | |====== | 8% | |====== | 9% | |======= | 10% | |======== | 11% | |======== | 12% | |========= | 13% | |========== | 14% | |=========== | 15% | |=========== | 16% | |============ | 17% | |============= | 18% | |============= | 19% | |============== | 20% | |=============== | 21% | |=============== | 22% | |================ | 23% | |================= | 24% | |================== | 25% | |================== | 26% | |=================== | 27% | |==================== | 28% | |==================== | 29% | |===================== | 30% | |====================== | 31% | |====================== | 32% | |======================= | 33% | |======================== | 34% | |========================= | 35% | |========================= | 36% | |========================== | 37% | |=========================== | 38% | |=========================== | 39% | |============================ | 40% | |============================= | 41% | |============================= | 42% | |============================== | 43% | |=============================== | 44% | |================================ | 45% | |================================ | 46% | |================================= | 47% | |================================== | 48% | |================================== | 49% | |=================================== | 50% | |==================================== | 51% | |==================================== | 52% | |===================================== | 53% | |====================================== | 54% | |======================================= | 55% | |======================================= | 56% | |======================================== | 57% | |========================================= | 58% | |========================================= | 59% | |========================================== | 60% | |=========================================== | 61% | |=========================================== | 62% | |============================================ | 63% | |============================================= | 64% | |============================================== | 65% | |============================================== | 66% | |=============================================== | 67% | |================================================ | 68% | |================================================ | 69% | |================================================= | 70% | |================================================== | 71% | |================================================== | 72% | |=================================================== | 73% | |==================================================== | 74% | |===================================================== | 75% | |===================================================== | 76% | |====================================================== | 77% | |======================================================= | 78% | |======================================================= | 79% | |======================================================== | 80% | |========================================================= | 81% | |========================================================= | 82% | |========================================================== | 83% | |=========================================================== | 84% | |============================================================ | 85% | |============================================================ | 86% | |============================================================= | 87% | |============================================================== | 88% | |============================================================== | 89% | |=============================================================== | 90% | |================================================================ | 91% | |================================================================ | 92% | |================================================================= | 93% | |================================================================== | 94% | |=================================================================== | 95% | |=================================================================== | 96% | |==================================================================== | 97% | |===================================================================== | 98% | |===================================================================== | 99% | |======================================================================| 100%
#> acceptance rate: 42%
#> sampling phase
#> | | | 0% | |= | 1% | |= | 2% | |== | 3% | |=== | 4% | |==== | 5% | |==== | 6% | |===== | 7% | |====== | 8% | |====== | 9% | |======= | 10% | |======== | 11% | |======== | 12% | |========= | 13% | |========== | 14% | |=========== | 15% | |=========== | 16% | |============ | 17% | |============= | 18% | |============= | 19% | |============== | 20% | |=============== | 21% | |=============== | 22% | |================ | 23% | |================= | 24% | |================== | 25% | |================== | 26% | |=================== | 27% | |==================== | 28% | |==================== | 29% | |===================== | 30% | |====================== | 31% | |====================== | 32% | |======================= | 33% | |======================== | 34% | |========================= | 35% | |========================= | 36% | |========================== | 37% | |=========================== | 38% | |=========================== | 39% | |============================ | 40% | |============================= | 41% | |============================= | 42% | |============================== | 43% | |=============================== | 44% | |================================ | 45% | |================================ | 46% | |================================= | 47% | |================================== | 48% | |================================== | 49% | |=================================== | 50% | |==================================== | 51% | |==================================== | 52% | |===================================== | 53% | |====================================== | 54% | |======================================= | 55% | |======================================= | 56% | |======================================== | 57% | |========================================= | 58% | |========================================= | 59% | |========================================== | 60% | |=========================================== | 61% | |=========================================== | 62% | |============================================ | 63% | |============================================= | 64% | |============================================== | 65% | |============================================== | 66% | |=============================================== | 67% | |================================================ | 68% | |================================================ | 69% | |================================================= | 70% | |================================================== | 71% | |================================================== | 72% | |=================================================== | 73% | |==================================================== | 74% | |===================================================== | 75% | |===================================================== | 76% | |====================================================== | 77% | |======================================================= | 78% | |======================================================= | 79% | |======================================================== | 80% | |========================================================= | 81% | |========================================================= | 82% | |========================================================== | 83% | |=========================================================== | 84% | |============================================================ | 85% | |============================================================ | 86% | |============================================================= | 87% | |============================================================== | 88% | |============================================================== | 89% | |=============================================================== | 90% | |================================================================ | 91% | |================================================================ | 92% | |================================================================= | 93% | |================================================================== | 94% | |=================================================================== | 95% | |=================================================================== | 96% | |==================================================================== | 97% | |===================================================================== | 98% | |===================================================================== | 99% | |======================================================================| 100%
#> acceptance rate: 44%
#> chain completed in 851.212754 seconds
#> total MCMC run-time: 851 secondsVisualising the Results
After running the model, we can visualise the results to compare the model’s predictions with the true synthetic data. This helps us ensure that the model correctly infers the impact of malaria on haemoglobin levels.
# Generate combined plot for non-censored data
combined_noncensored_plot <- generate_combined_plot(
model = run_noncensored_data,
fitted_data = uncensored_data_list$country_df,
true_data = synthetic_data_df,
site_inf_history = selected_inf_history
)In the left-hand plot dots show datapoints, the dashed lines show true underlying mean Hb by gestation and gravidity and thin equivalent lines show draws from the joint posterior for these means. On the right-hand plots thick orange lines show the simulated values and thin purple lines equivalent posterior draws.
Here we have run the mcmc for a much shorter duration than would be required for a final version but even in this shorter run we can the inferred and simulated relationships look reassuringly aligned.
