diff --git a/.github/workflows/build-website.yml b/.github/workflows/build-website.yml index b20d123a7..5b35f6cd8 100644 --- a/.github/workflows/build-website.yml +++ b/.github/workflows/build-website.yml @@ -71,7 +71,7 @@ jobs: R -q -e 'install.packages("renv", repos = "cloud.r-project.org"); renv::restore()' ls /home/runner/work/mlr3website/mlr3website/mlr-org/renv/library/R-4.2/x86_64-pc-linux-gnu git config --global user.email "quarto-github-actions-publish@example.com" - git config --global user.name "${{ env.RUNNER_NAME }}" + git config --global user.name "mlr-org" quarto publish gh-pages # - name: Install Packages diff --git a/mlr-org/_site/index.html b/mlr-org/_site/index.html deleted file mode 100644 index 1c1e9e0f7..000000000 --- a/mlr-org/_site/index.html +++ /dev/null @@ -1,655 +0,0 @@ - - - - - - - - - -<i class='bi bi-house-fill'></i> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
- -
- -
- - - - -
- - - - -
-

mlr(3): Machine Learning in R

-
-

An open-source collection of R packages providing a unified interface for machine learning in the R language.

-
- -

-
-
-
- - - -
-
A scientifically designed and easy to learn interface.
- -
-
-
- - - -
-
More than 100 connected machine learning algorithms.
- -
-
-
- - - -
-
Light on dependencies.
- -
-
-
- - - -
-
Convenient parallelization with the future package.
- -
-
-
- - - -
-
State-of-the-art optimization algorithms.
- -
-
-
- - - -
-
Dataflow programming with pipelines.
- -
-
-
-No matching items -
-
-
-

Resources

-
-
-
- - -GitHub - - - -Mattermost - - - -Book - - - -StackOverflow - -
-
-
-
-
-

Quick Start

-

The mlr3verse meta-package installs mlr3 and some of the most important extension packages:

-
-
install.packages("mlr3verse")
-
-
-
-

Examples

-
- -

-
-
-
- -
Basic machine learning
- -
-
    -
  • Learner
    • -
  • Task
    • -
  • Resampling
    • -
-
- -
-
-
- -
Hyperparameter tuning
- -
-
    -
  • Tuning
    • -
  • Tuning spaces
    • -
  • Nested resampling
    • -
-
- -
-
-
-No matching items -
-
- - - -
- -
- -
- - - - - \ No newline at end of file diff --git a/mlr-org/_site/search.json b/mlr-org/_site/search.json deleted file mode 100644 index 2c60729af..000000000 --- a/mlr-org/_site/search.json +++ /dev/null @@ -1,2368 +0,0 @@ -[ - { - "objectID": "tasks.html", - "href": "tasks.html", - "title": "", - "section": "", - "text": "The mlr3 packages also ship with some data sets, readily usable as Task objects. The goal of these tasks is to quickly demonstrate the capabilities of the packages.\n\n\n\n\n\n\n\n\n\nCreate a classification task from the data set in the palmerpenguins package.\n\nlibrary(\"mlr3verse\")\n\n# create a task\ntask = tsk(\"breast_cancer\")\ntask\n\n (683 x 10): Wisconsin Breast Cancer\n* Target: class\n* Properties: twoclass\n* Features (9):\n - ord (9): bare_nuclei, bl_cromatin, cell_shape, cell_size,\n cl_thickness, epith_c_size, marg_adhesion, mitoses, normal_nucleoli\n\n# get the dimensions\nc(task$nrow, task$ncol)\n\n[1] 683 10\n\n# check for missing values\ntask$missings()\n\n class bare_nuclei bl_cromatin cell_shape cell_size \n 0 0 0 0 0 \n cl_thickness epith_c_size marg_adhesion mitoses normal_nucleoli \n 0 0 0 0 0 \n\n# plot class frequencies\nautoplot(task)" - }, - { - "objectID": "measures.html", - "href": "measures.html", - "title": "", - "section": "", - "text": "Measures\nMeasures operate on Prediction objects generated by learners. They quantify the prediction by comparing prediction with ground truth. The Measure objects provide an abstraction for a plethora of performance measures." - }, - { - "objectID": "learners.html", - "href": "learners.html", - "title": "", - "section": "", - "text": "To keep the dependencies on other packages reasonable, the base package mlr3 only ships with with regression and classification trees from the rpart package and some learners for debugging. A subjective selection of implementations for essential ML algorithms can be found in mlr3learners package. Survival learners are provided by mlr3proba, cluster learners via mlr3cluster. Additional learners, including some learners which are not yet to be considered stable or which are not available on CRAN, are connected via the mlr3extralearners package.\n\n\n\n\n\n\n\n\n\nFit a classification tree on the Wisconsin Breast Cancer Data Set and predict on left-out observations.\n\nlibrary(\"mlr3verse\")\n\n# retrieve the task\ntask = tsk(\"breast_cancer\")\n\n# split into two partitions\nsplit = partition(task)\n\n# retrieve a learner\nlearner = lrn(\"classif.rpart\", keep_model = TRUE, predict_type = \"prob\")\n\n# fit decision tree\nlearner$train(task, split$train)\n\n# access learned model\nlearner$model\n\nn= 457 \n\nnode), split, n, loss, yval, (yprob)\n * denotes terminal node\n\n1) root 457 160 benign (0.35010941 0.64989059) \n 2) cell_size=4,5,6,7,8,9,10 146 8 malignant (0.94520548 0.05479452) *\n 3) cell_size=1,2,3 311 22 benign (0.07073955 0.92926045) \n 6) bare_nuclei=5,6,7,8,9,10 21 4 malignant (0.80952381 0.19047619) *\n 7) bare_nuclei=1,2,3,4 290 5 benign (0.01724138 0.98275862) *\n\n# predict on data frame with new data\npredictions = learner$predict_newdata(task$data(split$test))\n\n# predict on subset of the task\npredictions = learner$predict(task, split$test)\n\n# inspect predictions\npredictions\n\n for 226 observations:\n row_ids truth response prob.malignant prob.benign\n 1 benign benign 0.01724138 0.98275862\n 8 benign benign 0.01724138 0.98275862\n 10 benign benign 0.01724138 0.98275862\n--- \n 654 malignant malignant 0.94520548 0.05479452\n 666 malignant malignant 0.94520548 0.05479452\n 676 malignant malignant 0.94520548 0.05479452\n\npredictions$score(msr(\"classif.auc\"))\n\nclassif.auc \n 0.9529837 \n\nautoplot(predictions, type = \"roc\")" - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html", - "title": "mlr Workshop 2017", - "section": "", - "text": "In 2017, we are hosting the workshop at LMU Munich. The workshop will run from 6 March to 10 March 2017 (potentially including the sunday before and the saturday at the end), hosted by the Ludwig-Maximilians-University Munich.\nImportant Dates:\n\nAddress: Geschwister-Scholl-Platz 1, Room: M203.\nStart: 6th of March: 10:00 AM.\n\nIt is also possible to arrive on Saturday or Sunday, as we already have the rooms and are able to work there. But this is totally optional and the official workshop starts on Monday. Same thing for the Saturday after the workshop." - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#why", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#why", - "title": "mlr Workshop 2017", - "section": "Why?", - "text": "Why?\nThe mlr developer team is quite international: Germany, USA, Canada. The time difference between these countries sometimes makes it hard to communicate and develop new features. The idea for this workshop or sprint was to have the possibility to talk about the project status, future and structure, exterminate imperishable bugs and start developing some fancy features." - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#what-is-the-target-audience-can-i-join", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#what-is-the-target-audience-can-i-join", - "title": "mlr Workshop 2017", - "section": "What is the target audience / Can I join?", - "text": "What is the target audience / Can I join?\nThe workshop is mainly geared towards the already existing crowd of developers, but it might also be a perfect opportunity to join the team - if you want to help. We are always looking for competent persons to collaborate with. If you are interested, please register in the following form, and we are looking forward to seeing you in Munich! Join us for the excellent team and the nice Bavarian beer and food in Munich!\nRegister here" - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#sponsors", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#sponsors", - "title": "mlr Workshop 2017", - "section": "Sponsors", - "text": "Sponsors\nWe want to thank all our sponsors. Without them this workshop would not be possible." - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#timetable-and-schedule", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#timetable-and-schedule", - "title": "mlr Workshop 2017", - "section": "Timetable and schedule", - "text": "Timetable and schedule\nYou can find the workshop schedule here" - }, - { - "objectID": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#results-of-the-previous-workshop", - "href": "posts/2017-02-13-mlr-workshop/2017-02-13-mlr-workshop.html#results-of-the-previous-workshop", - "title": "mlr Workshop 2017", - "section": "Results of the previous workshop", - "text": "Results of the previous workshop\nOur last workshop in 2016 was in Palermo, Italy. Twelve people from the developer team met from the 8th to 15th of August to work on and with mlr and it was more like a sprint, as our core developers meet to get stuff done. We closed a lot of issues and developed new features that we will release with version 2.10 of mlr. Thanks to all participants: Giuseppe Casalicchio, Janek Thomas, Xudong Sun, Jakob Bossek, Bernd Bischl, Jakob Richter, Michel Lang, Philipp Probst, Julia Schiffner, Lars Kotthoff, Zachary Jones, Pascal Kerschke! We also head a great time in a great city aside from the workshop, where we have been travelling around the city for sightseeing and enjoying the beach and nice food of Palermo." - }, - { - "objectID": "posts/2017-03-28-multilabel-classification-with-mlr/2017-03-28-multilabel-classification-with-mlr.html", - "href": "posts/2017-03-28-multilabel-classification-with-mlr/2017-03-28-multilabel-classification-with-mlr.html", - "title": "Multilabel Classification with mlr", - "section": "", - "text": "Multilabel classification has lately gained growing interest in the research community. We implemented several methods, which make use of the standardized mlr framework. Every available binary learner can be used for multilabel problem transformation methods. So if you’re interested in using several multilabel algorithms and want to know how to use them in the mlr framework, then this post is for you!\n\n1) Introduction to multilabel classification\nFirst, let me introduce you to multilabel classification. This is a classification problem, where every instance can have more than one label. Let’s have a look at a typical multilabel dataset (which I, of course, download from the OpenML server):\n\nlibrary(mlr)\nlibrary(OpenML)\nsetOMLConfig(apikey = \"c1994bdb7ecb3c6f3c8f3b35f4b47f1f\") # api key\noml.id = listOMLDataSets(tag = \"2016_multilabel_r_benchmark_paper\")$data.id\nscene = getOMLDataSet(data.id = oml.id[8])\ntarget = scene$target.features\nfeats = setdiff(colnames(scene$data), target)\n\n\nhead(scene$data[, c(feats[1], feats[2], target)])\n## Att1 Att2 Beach Sunset FallFoliage Field Mountain Urban\n## 0 0.646467 0.666435 TRUE FALSE FALSE FALSE TRUE FALSE\n## 1 0.770156 0.767255 TRUE FALSE FALSE FALSE FALSE TRUE\n## 2 0.793984 0.772096 TRUE FALSE FALSE FALSE FALSE FALSE\n## 3 0.938563 0.949260 TRUE FALSE FALSE FALSE FALSE FALSE\n## 4 0.512130 0.524684 TRUE FALSE FALSE FALSE FALSE FALSE\n## 5 0.824623 0.886845 TRUE FALSE FALSE FALSE FALSE FALSE\n\nHere I took the scene dataset, where the features represent color information of pictures and the targets could be objects like beach, sunset, and so on.\nAs you can see above, one defining property of a multilabel dataset is, that the target variables (which are called labels) are binary. If you want to use your own data set, make sure to encode these variables in logical, where TRUE indicates the relevance of a label.\nThe basic idea behind many multilabel classification algorithms is to make use of possible correlation between labels. Maybe a learner is very good at predicting label 1, but rather bad at predicting label 2. If label 1 and label 2 are highly correlated, it may be beneficial to predict label 1 first and use this prediction as a feature for predicting label 2.\nThis approach is the main concept behind the so called problem transformation methods. The multilabel problem is transformed into binary classification problems, one for each label. Predicted labels are used as features for predicting other labels.\nWe implemented the following problem transformation methods:\n\nClassifier chains\nNested stacking\nDependent binary relevance\nStacking\n\nHow these methods are defined, can be read in the mlr tutorial or in more detail in our paper. Enough theory now, let’s apply these methods on our dataset.\n\n\n2) Let’s Train and Predict!\nFirst we need to create a multilabel task.\n\nset.seed(1729)\ntarget\n## [1] \"Beach\" \"Sunset\" \"FallFoliage\" \"Field\" \"Mountain\" \n## [6] \"Urban\"\nscene.task = makeMultilabelTask(data = scene$data, target = target)\n\nWe set a seed, because the classifier chain wrapper uses a random chain order. Next, we train a learner. I chose the classifier chain approach together with a decision tree for the binary classification problems.\n\nbinary.learner = makeLearner(\"classif.rpart\")\nlrncc = makeMultilabelClassifierChainsWrapper(binary.learner)\n\nNow let’s train and predict on our dataset:\n\nn = getTaskSize(scene.task)\ntrain.set = seq(1, n, by = 2)\ntest.set = seq(2, n, by = 2)\n\nscene.mod.cc = train(lrncc, scene.task, subset = train.set)\nscene.pred.cc = predict(scene.mod.cc, task = scene.task, subset = test.set)\n\nWe also implemented common multilabel performance measures. Here is a list with available multilabel performance measures:\n\nlistMeasures(\"multilabel\")\n## [1] \"featperc\" \"multilabel.tpr\" \"multilabel.hamloss\" \n## [4] \"multilabel.subset01\" \"timeboth\" \"timetrain\" \n## [7] \"timepredict\" \"multilabel.ppv\" \"multilabel.f1\" \n## [10] \"multilabel.acc\"\n\nHere is how the classifier chains method performed:\n\nperformance(scene.pred.cc, measures = list(multilabel.hamloss,\n multilabel.subset01, multilabel.f1, multilabel.acc))\n## multilabel.hamloss multilabel.subset01 multilabel.f1 multilabel.acc \n## 0.1318925 0.4887781 0.5785259 0.5613743\n\n\n\n3) Comparison Binary Relevance vs. Classifier Chains\nNow let’s see if it can be beneficial to use predicted labels as features for other labels. Let us compare the performance of the classifier chains method with the binary relevance method (this method does not use predicted labels as features).\n\nlrnbr = makeMultilabelBinaryRelevanceWrapper(binary.learner)\n\nscene.mod.br = train(lrnbr, scene.task, subset = train.set)\nscene.pred.br = predict(scene.mod.br, task = scene.task, subset = test.set)\n\nperformance(scene.pred.br, measures = list(multilabel.hamloss,\n multilabel.subset01, multilabel.f1, multilabel.acc))\n## multilabel.hamloss multilabel.subset01 multilabel.f1 multilabel.acc \n## 0.1305071 0.5719036 0.5357163 0.5083818\n\nAs can be seen here, it could indeed make sense to use more elaborate methods for multilabel classification, since classifier chains beat the binary relevance methods in all of these measures (Note, that hamming loss and subset01 are loss measures!).\n\n\n4) Resampling\nHere I’ll show you how to use resampling methods in the multilabel setting. Resampling methods are key for assessing the performance of a learning algorithm. To read more about resampling, see the page on our tutorial.\nFirst, we need to define a resampling strategy. I chose subsampling, which is also called Monte-Carlo cross-validation. The dataset is split into training and test set at a predefined ratio. The learner is trained on the training set, the performance is evaluated with the test set. This whole process is repeated many times and the performance values are averaged. In mlr this is done the following way:\n\nrdesc = makeResampleDesc(\"Subsample\", iters = 10, split = 2/3)\n\nNow we can choose a measure, which shall be resampled. All there is left to do is to run the resampling:\n\nr = resample(lrncc, scene.task, rdesc, measures = multilabel.subset01)\n\n\nr\n## Resample Result\n## Task: scene$data\n## Learner: multilabel.classifierChains.classif.rpart\n## Aggr perf: multilabel.subset01.test.mean=0.4912827\n## Runtime: 6.32109\n\nIf you followed the mlr tutorial or if you are already familiar with mlr, you most likely saw, that using resampling in the multilabel setting isn’t any different than generally using resampling in mlr. Many methods, which are available in mlr, like preprocessing, tuning or benchmark experiments can also be used for multilabel datasets and the good thing here is: the syntax stays the same!" - }, - { - "objectID": "posts/2016-08-23-how-to-win-a-drone-in-20-lines-of-r-code/2016-08-23-how-to-win-a-drone-in-20-lines-of-r-code.html", - "href": "posts/2016-08-23-how-to-win-a-drone-in-20-lines-of-r-code/2016-08-23-how-to-win-a-drone-in-20-lines-of-r-code.html", - "title": "How to win a drone in 20 lines of R code", - "section": "", - "text": "Or a less clickbaity title: Model based optimization of machine learning models with mlr and mlrMBO.\nI recently participated in the #TEFDataChallenge a datathon organized by Wayra. The first price was a drone for every team member, which is a pretty awesome price.\nSo what exactly is a datathon?\n\nAt a datathon multiple teams consisting out of computer engineers, data scientists and experts from other fields come together to work on a case for 24 hours straight, where the purpose is to study and analyze data related to the case. Methods well known in AI will be used. A datathon is derived from a so-called hackathon, where in this case the focus is more on data rather than on innovation. Although with the upcoming field of big data bigger and more complicated problems can be solved, there is still a lack of real data scientists. Therefore a datathon is the solution: In a fun and challenging way people are triggered to come up with the best ideas! The Team with the most outstanding patterns or the best predictive models will win the datathon!\n\nSource\nI already took part in some hackathons, but never in a datathon so this was a bit new for me. But only 24 hours to create a prediction model as well as an interesting visualization seems quite hard.\nSince I signed a NDA, I won’t talk about the data used in the challenge. Only so much (that’s the information you can find on their website):\n\nThe aim is to build a prediction model for customer churn, a model to predict if a customer will or will not cancel a mobile contract.\n\nThis is a typical binary classification problem. Fortunately the data was already nicely preprocessed. No missing values or data garbage, they even aggregated the data set already so that only one observation per costumer remained. This makes the problem a bit easier: find the best prediction model in a short amount of time, but most importantly: We don’t want to spend hours coding it, so that we can use most of our time to create an awesome visualization and create/scrape additional features.\nTypically in these kind of challenges stacking results in the winning model. But this problematic in this case because\n\nStacking multiple models takes an extremely long time and/or huge computation power. We only had one EC2 instance.\nThe judges wanted to have simple models, the simpler the model the better, while still having extremely good prediction accuracy.\n\nSo what to do when we cannot stack a hugely complected model ensemble? We use gradient boosting with trees. This has multiple advantages, we have a (rather) simple model with good prediction accuracy, we can handle categorical variables with large number of classes and we get variable importance measures which can be used in the visualization of the model. (We can also create partial dependence plots to gain even more insights in the effects). We use xgboost which is currently the fastest implementation for gradient boosting with trees.\nHere is our learner definition with the param-set we want to optimize.\n\nlibrary(mlr)\nlrn = makeLearner(\"classif.xgboost\", eval_metric = \"auc\",\n predict.type = \"prob\")\n\nps = makeParamSet(\n makeIntegerParam(\"nrounds\", lower = 200, upper = 2500, default = 200),\n makeNumericParam(\"eta\", lower = -7, upper = -5, default = -6,\n trafo = function(x) 2^x),\n makeIntegerParam(\"max_depth\", lower = 3, upper = 15, default = 3),\n makeNumericParam(\"colsample_bytree\", lower = 0.3, upper = 1,\n default = 0.6),\n makeNumericParam(\"subsample\", lower = 0.3, upper = 1, default = 0.6)\n)\n\nOverall we have 5 parameters (number of trees, learning rate, tree depth, bagging fraction and sub-sampling fraction) that we vary to find our optimal model. For standard optimization techniques like grid search this is already a real problem because the numbers of points to search increases exponentially with every additional hyper-parameter.\nSince all hyper-parameter are numeric it would be possible to use evolutionary algorithms like cmaes. The problem with that is, that these methods generally take a large number of function evaluation to find the optimal model, but we don’t have too much time and (cross-validated) model fits are quite expensive.\nThis is basically the perfect situation for model based optimization. We fit a surrogate model over the space of hyper-parameters and search promising points. Having only numeric hyper-parameter makes the optimization even nicer because we can use a Gaussian process as our surrogate model. In the figure you can see an example of model based optimization in 1 dimension. The upper figure shows evaluated points, the estimated model (dashed line) and its variance (gray area). The lower picture shows how interesting every possible point is for future exploration of the space. For an in-depth introduction to model-based optimization you can read Jones(1998).\n\n\n\n\n\n\n\n\n\n\n\n\nWe use mlrMBO as a general black-box optimization toolkit with is already nicely connected to mlr. The optimization definition looks like this:\n\nlibrary(mlrMBO)\nlibrary(parallelMap)\ntask = makeClassifTask(data = data, target = \"churn\")\nmbo.ctrl = makeMBOControl(save.on.disk.at = c(0, 5, 10, 20, 50, 75, 85, 95))\nmbo.ctrl = setMBOControlTermination(mbo.ctrl, iters = 100)\nsurrogate.lrn = makeLearner(\"regr.km\", predict.type = \"se\")\nctrl = mlr:::makeTuneControlMBO(learner = surrogate.lrn,\n mbo.control = mbo.ctrl)\n\nparallelStartMulticore(cpus = 10L)\nres.mbo = tuneParams(lrn, task, cv10, par.set = ps, control = ctrl,\n show.info = TRUE, measures = auc)\nparallelStop()\n\nSome notes on this:\n\nWe save our model (actually the optimization path) at different iterations of the process, which is quite useful if anything crashes.\nWe do 100 sequential iterations after the initial design (for which we used the default setting).\nWe use kriging as our surrogate model. Actually we don’t even need to specify this, since it is the default surrogate model of mlrMBO.\nparallelMap is used to parallelize the cross-validation over all 10 available cores on the EC2 instance.\n\nThe resulting model was in the end the best model on the hidden test set and even beat a large stacking ensemble of multiple boosting models, random forests and deep neural networks.\nOne other team used a quite similar approach, but instead of model based optimization they used irace to tune a gradient boosting model. This can also be done in mlr quite easily:\n\nctrl = makeTuneControlIrace(n.instances = 200L)\nparallelStartMulticore(cpus = 10L)\nres.irace = tuneParams(lrn, task, cv10, par.set = ps, control = ctrl,\n show.info = TRUE, measures = auc)\nparallelStop()" - }, - { - "objectID": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html", - "href": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html", - "title": "mlr3 Package Updates - Q3/2021", - "section": "", - "text": "Due to the high amount of packages in the mlr3 ecosystem, it is hard to keep up with the latest changes across all packages. This post tries to tackle this issue by listing all release notes of the packages most recent releases in the last quarter. Note that only CRAN packages are listed here and the sort order is alphabetically.\nInterval: 2021-07-01 - 2021-10-01" - }, - { - "objectID": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#breaking", - "href": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#breaking", - "title": "mlr3 Package Updates - Q3/2021", - "section": "Breaking", - "text": "Breaking\n\nautoplot(): removed argument crs. The CRS is now inferred from the supplied Task. Setting a different CRS than the task might lead to spurious issues and the initial idea of changing the CRS for plotting to have proper axes labeling does not apply (anymore) (#144)" - }, - { - "objectID": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#features", - "href": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#features", - "title": "mlr3 Package Updates - Q3/2021", - "section": "Features", - "text": "Features\n\nAdded autoplot() support for ResamplingCustomCV (#140)" - }, - { - "objectID": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#bug-fixes", - "href": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#bug-fixes", - "title": "mlr3 Package Updates - Q3/2021", - "section": "Bug fixes", - "text": "Bug fixes\n\n\"spcv_block\": Assert error if folds > 2 when selection = \"checkerboard\" (#150)\nFixed row duplication when creating TaskRegrST tasks from sf objects (#152)" - }, - { - "objectID": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#miscellaneous", - "href": "posts/2021-09-29-mlr3-package-updates-q32021/mlr3-package-updates-q32021.html#miscellaneous", - "title": "mlr3 Package Updates - Q3/2021", - "section": "Miscellaneous", - "text": "Miscellaneous\n\nUpgrade tests to {vdiffr} 1.0.0\nAdd {rgdal} to suggests and required it in \"spcv_block\" since it is required in {blockCV} >= 2.1.4 and {sf} >= 1.0" - }, - { - "objectID": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html", - "href": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html", - "title": "The Cross-Validation - Train/Predict Misunderstanding", - "section": "", - "text": "Over the past years I’ve seen multiple posts on Stackoverflow and our GitHub issues which suffer from a conceptual misunderstanding: cross-validation (CV) vs. train/predict.\nBecause train/predict is an essential part of cross-validation, the point might not be so obvious. I’ll try to make it more clear by providing some exemplary questions:\n\n“I’ve done cross-validation. How do I decided which model to use for prediction?”\n“I’ve used the resample() function. How do I decided which hyperparameters are best?”\n“I’ve benchmarked some algorithms. How do I find the most important features?”\n\nAll of these questions have a common problem: users trying to extract something out of the CV to work with it afterwards. They want to use the “best model” or find the “most important features” or similar. The thoughtful reader might already infer by now that doing so is probably problematic.\nAnd yes, it is. One should not try to extract a single component (be it a model, a set of hyperparameters or a set of predictors) out of a CV." - }, - { - "objectID": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#what-happens-in-a-cv-stays-in-a-cv", - "href": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#what-happens-in-a-cv-stays-in-a-cv", - "title": "The Cross-Validation - Train/Predict Misunderstanding", - "section": "What happens in a CV, stays in a CV", - "text": "What happens in a CV, stays in a CV\nHere’s why:\nEvery model fit and evaluation in a CV happens on a subset of the main dataset.\n\nThe dataset is split by a specific method (e.g. randomly) into pre-selected partitions (e.g. 5).\n(Optional) Optimal hyperparameters are searched and feature selection is performed (in another inner CV cycle)\nThe model is fit on the training set and predicts on the test set\nThe performance of this prediction is evaluated (because “truth” (= test set) is known)\n\nThis is done multiple times. Every time, the dataset is different. Every time, different hyperparameters are found. Every time - ok you got it by now.\nNone of these training/test set combinations represent overall “the best” choice - they only operate in their specific data split setting. There is also no way to find a model (or similar) within a CV with respect to these criteria.\nThe main reason for this is that in all cases the fitted model was trained on only a subset of the data available. This was done to evaluate the performance on a subset of the data - because “truth” is known for the hold back data. Otherwise there would be no need to hide precious data from model fitting." - }, - { - "objectID": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#train-predict", - "href": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#train-predict", - "title": "The Cross-Validation - Train/Predict Misunderstanding", - "section": "Train & Predict", - "text": "Train & Predict\nThe main purpose of fitting a model is make predictions with it. For this, you want to use all available data to fit the most robust model possible. And this is exactly what you should do: take all your data, optimize your hyperparameters, eventually conduct feature selection, and then fit the model.\nYes, do it again - by using the train() and predict() functions (and their tuning wrappers/pipelines) directly. Do not use resample() or benchmark() - these are for CV purposes!\nThen, take this one model and predict into unknown space. In this scenario, you cannot know how good your predictions will be because there is no “truth” to evaluate against. But this is perfectly fine. This is why a CV was done (beforehand): to have a somewhat unbiased estimate of how your model will perform under different conditions. You can also analyse the hyperparameters or evaluate the results of a feature selection from this model1.\nBe careful though: your final model which you fit and predicted into unknown space might exactly have this performance - but it might also be completely off. You will never find out unless you eventually collect data at some point which you can compare against the predictions made by your model. You can only state that, with a given uncertainty, your model will have a performance of X when predicting." - }, - { - "objectID": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#summary", - "href": "posts/2020-12-20-the-cross-validation-trainpredict-misunderstanding/the-cross-validation-trainpredict-misunderstanding.html#summary", - "title": "The Cross-Validation - Train/Predict Misunderstanding", - "section": "Summary", - "text": "Summary\nTo make if fully clear again: CV and train/predict are two separate things. Think of them as two different buckets with no relation to each other.\n\nCV is done to get an estimate of a model’s performance.\nTrain/predict is done to create the final predictions (which your boss might use to make some decisions on).\n\nCV is used to explain the performance of your fitted model (which is a single fit of the chosen algorithm on all of your data points)." - }, - { - "objectID": "posts/2019-05-06-mlr-example-for-drake/2019-05-06-mlr-example-for-drake.html", - "href": "posts/2019-05-06-mlr-example-for-drake/2019-05-06-mlr-example-for-drake.html", - "title": "mlr + drake: Reproducible machine-learning workflow management", - "section": "", - "text": "You may have heard about the drake package. It got a lot attention recently in the R community because it simplifies reproducible workflow management. This comes especially handy for large projects which have hundreds of intermediate steps. Built-in High-Performance-Cluster (HPC) support and graph visualization are just two goodies that come on top of the basic functionality.\ndrake is able to track changes in your intermediate targets. This means once you change something in your early workflow pipeline, drake will automatically update all follow-up objects that might be affected by this change. The following tweet wraps the struggle of keeping track of dependencies in a research project in an simple picture:\n\n\nSave me from myself and having to remember all this when files change pic.twitter.com/hVeSFQOimj\n\n— Dr. Brianna McHorse 🏳️‍🌈 (@fossilosophy) 21. Februar 2018\n\n\nThe maintainer of drake, Will Landau (@wlandau) is extremely responsive and has also written one of the most extensive and detailed manuals that exist in the R package jungle.\nIf you have installed drake, you can start right away with one of the built-in examples.\ndrake::drake_example(\"mlr-slurm\")\nAt the time of writing, there are 17(!) examples that you can choose from. One of the newest shows how to use mlr with drake on a HPC.\nMachine-Learning projects/tasks interact especially well with the drake idea since you can easily create large comparison matrices using different algorithms / hyperparameter settings. At the same time drake can sent these settings in parallel to a HPC for you, simplifying your modeling tasks a lot." - }, - { - "objectID": "posts/2017-05-16-shinymlr/2017-05-16-shinymlr.html", - "href": "posts/2017-05-16-shinymlr/2017-05-16-shinymlr.html", - "title": "shinyMlr", - "section": "", - "text": "shinyMlr is a web application, built with the R-package “shiny” that provides a user interface for mlr. By wrapping the main functionalities of mlr into our app, as well as implementing additional features for data visualisation and data preprocessing, we built a widely usable application for your day to day machine learning tasks, which we would like to present to you today.\nStefan and me started working on this project late summer 2016 as part of a practical course we attended for our Master’s program. We enjoyed the work on this project and will continue to maintain and extend our app in the future. However, after almost one year of work our application got a versatile tool and it is time to present it to a broader audience. To introduce you to the workflow and main features of our app, we uploaded a video series to our youtube channel. The videos are little tutorials that illustrate the workflow in form of a use case: We used the titanic data set from the kaggle competition as example data to show you step by step how it can be analyzed with our application.\nThe first video gives a small introduction and shows you how data can be imported: Tutorial 1\nIn the next tutorial you will learn how to visualise your data and preprocess it: Tutorial 2\nThe third and fourth screencasts show you how to create your task and how to construct and modify our built-in learning algorithms: Tutorial 3, Tutorial 4\nThe fifth part of our tutorials shows you how to tune your learners to find suitable parameter settings for your given training set: Tutorial 5\nThe sixth video gives you detailed information on how to actually train models on your task, predict on new data and plot model diagnostic and prediction plots: Tutorial 6\nThe seventh video runs a benchmark experiment, to show you how to compare different learners in our application: Tutorial 7\nThe last tutorial briefly demonstrates how to render an interactive report from your analysis done with our app: Tutorial 8\nI hope you enjoyed watching the videos and learned how to make use of our application. If you like working with our app please leave us a star and follow us on github" - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html", - "title": "mlr3 package Updates - Q4/2021", - "section": "", - "text": "Due to the high amount of packages in the mlr3 ecosystem, it is hard to keep up with the latest changes across all packages. This posts gives an overview by listing the recent release notes of mlr3 packages from the last quarter. Note that only CRAN packages are listed here and the sort order is alphabetically.\nInterval: 2021-10-01 - 2021-12-31" - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3-0.13.0", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3-0.13.0", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3 0.13.0", - "text": "mlr3 0.13.0\nDescription: Machine Learning in R - Next Generation\n\nLearners which are capable of resuming/continuing (e.g., learner (classif|regr|surv).xgboost with hyperparameter nrounds updated) can now optionally store a stack of trained learners to be used to hotstart their training. Note that this feature is still somewhat experimental. See HotstartStack and #719.\nNew measures to score similarity of selected feature sets: sim.jaccard (Jaccard Index) and sim.phi (Phi coefficient) (#690).\npredict_newdata() now also supports DataBackend as input.\nNew function install_pkgs() to install required packages. This generic works for all objects with a packages field as well as ResampleResult and BenchmarkResult (#728).\nNew learner regr.debug for debugging.\nNew Task method $set_levels() to control how data with factor columns is returned, independent of the used DataBackend.\nMeasures now return NA if prerequisite are not met (#699). This allows to conveniently score your experiments with multiple measures having different requirements.\nFeature names may no longer contain the special character %." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3benchmark-0.1.3", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3benchmark-0.1.3", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3benchmark 0.1.3", - "text": "mlr3benchmark 0.1.3\nDescription: Analysis and Visualisation of Benchmark Experiments\n\nFix README.\nFix for PMCMRplus." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3db-0.4.2", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3db-0.4.2", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3db 0.4.2", - "text": "mlr3db 0.4.2\nDescription: Data Base Backend for ‘mlr3’\n\nCompatibility fixes with new duckdb version." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3learners-0.5.1", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3learners-0.5.1", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3learners 0.5.1", - "text": "mlr3learners 0.5.1\nDescription: Recommended learners for mlr3\n\neval_metric() is now explicitly set for xgboost learners to silence a deprecation warning.\nImproved how the added hyperparameter mtry.ratio is converted to mtry to simplify tuning.\nMultiple updates to hyperparameter sets." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3pipelines-0.4.0", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3pipelines-0.4.0", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3pipelines 0.4.0", - "text": "mlr3pipelines 0.4.0\nDescription: Preprocessing Operators and Pipelines for ‘mlr3’\n\nNew operator %>>!% that modifies Graphs in-place.\nNew methods chain_graphs(), concat_graphs(), Graph$chain() as alternatives for %>>% and %>>!%.\nNew methods pos() and ppls() which create lists of PipeOps/Graphs and can be seen as “plural” forms of po() and ppl().\npo() S3-method for PipeOp class that clones a PipeOp object and optionally modifies its attributes.\nGraph$add_pipeop() now clones the PipeOp being added.\nDocumentation: Clarified documentation about cloning of input arguments in several places.\nPerformance enhancements for Graph concatenation.\nMore informative error outputs.\nNew attribute graph_model GraphLearner class, which gets the trained graph.\nas_learner() S3-method for PipeOp class that turns wraps a PipeOp in a Graph and turns that into a Learner.\nChanged PipeOps:\n\nPipeOpHistBin: renamed ‘bins’ Param to ‘breaks’\nPipeOpImputeHist: fix handling of integer features spanning the entire represented integer range\nPipeOpImputeOOR: fix handling of integer features spanning the entire represented integer range\nPipeOpProxy: Avoid unnecessary clone\nPipeOpScale: Performance improvement" - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3proba-0.4.2", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3proba-0.4.2", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3proba 0.4.2", - "text": "mlr3proba 0.4.2\nDescription: Probabilistic Supervised Learning for ‘mlr3’\n\nPatch for linux." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3spatial-0.1.0", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3spatial-0.1.0", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3spatial 0.1.0", - "text": "mlr3spatial 0.1.0\nDescription: Support for Spatial Objects Within the ‘mlr3’ Ecosystem\n\nInitial release." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3tuningspaces-0.0.1", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3tuningspaces-0.0.1", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3tuningspaces 0.0.1", - "text": "mlr3tuningspaces 0.0.1\nDescription: Search Spaces for Hyperparameter Tuning\n\nInitial release." - }, - { - "objectID": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3viz-0.5.7", - "href": "posts/2022-01-03-mlr3-package-updates-q42021/mlr3-package-updates-q42021.html#mlr3viz-0.5.7", - "title": "mlr3 package Updates - Q4/2021", - "section": "mlr3viz 0.5.7", - "text": "mlr3viz 0.5.7\nDescription: Visualizations for ‘mlr3’\n\nCompatibility fix for testthat." - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "", - "text": "Machine learning models repeatedly outperform interpretable, parametric models like the linear regression model. The gains in performance have a price: The models operate as black boxes which are not interpretable.\nFortunately, there are many methods that can make machine learning models interpretable. The R package iml provides tools for analysing any black box machine learning model:\nThis blog post shows you how to use the iml package to analyse machine learning models. While the mlr package makes it super easy to train machine learning models, the iml package makes it easy to extract insights about the learned black box machine learning models.\nIf you want to learn more about the technical details of all the methods, read the Interpretable Machine Learning book.\nLet’s explore the iml-toolbox for interpreting an mlr machine learning model with concrete examples!" - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#data-boston-housing", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#data-boston-housing", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "Data: Boston Housing", - "text": "Data: Boston Housing\nWe’ll use the MASS::Boston dataset to demonstrate the abilities of the iml package. This dataset contains median house values from Boston neighbourhoods.\n\ndata(\"Boston\", package = \"MASS\")\nhead(Boston)\n#> crim zn indus chas nox rm age dis rad tax ptratio black lstat\n#> 1 0.00632 18 2.31 0 0.538 6.575 65.2 4.0900 1 296 15.3 396.90 4.98\n#> 2 0.02731 0 7.07 0 0.469 6.421 78.9 4.9671 2 242 17.8 396.90 9.14\n#> 3 0.02729 0 7.07 0 0.469 7.185 61.1 4.9671 2 242 17.8 392.83 4.03\n#> 4 0.03237 0 2.18 0 0.458 6.998 45.8 6.0622 3 222 18.7 394.63 2.94\n#> 5 0.06905 0 2.18 0 0.458 7.147 54.2 6.0622 3 222 18.7 396.90 5.33\n#> 6 0.02985 0 2.18 0 0.458 6.430 58.7 6.0622 3 222 18.7 394.12 5.21\n#> medv\n#> 1 24.0\n#> 2 21.6\n#> 3 34.7\n#> 4 33.4\n#> 5 36.2\n#> 6 28.7" - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#fitting-the-machine-learning-model", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#fitting-the-machine-learning-model", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "Fitting the machine learning model", - "text": "Fitting the machine learning model\nFirst we train a randomForest to predict the Boston median housing value:\n\nlibrary(\"mlr\")\ndata(\"Boston\", package = \"MASS\")\n\n# create an mlr task and model\ntsk = makeRegrTask(data = Boston, target = \"medv\")\nlrn = makeLearner(\"regr.randomForest\", ntree = 100)\nmod = train(lrn, tsk)" - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#using-the-iml-predictor-container", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#using-the-iml-predictor-container", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "Using the iml Predictor container", - "text": "Using the iml Predictor container\nWe create a Predictor object, that holds the model and the data. The iml package uses R6 classes: New objects can be created by calling Predictor$new(). Predictor works best with mlr models (WrappedModel-class), but it is also possible to use models from other packages.\n\nlibrary(\"iml\")\nX = Boston[which(names(Boston) != \"medv\")]\npredictor = Predictor$new(mod, data = X, y = Boston$medv)" - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#feature-importance", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#feature-importance", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "Feature importance", - "text": "Feature importance\nWe can measure how important each feature was for the predictions with FeatureImp. The feature importance measure works by shuffling each feature and measuring how much the performance drops. For this regression task we choose to measure the loss in performance with the mean absolute error (‘mae’); another choice would be the mean squared error (‘mse’).\nOnce we created a new object of FeatureImp, the importance is automatically computed. We can call the plot() function of the object or look at the results in a data.frame.\n\nimp = FeatureImp$new(predictor, loss = \"mae\")\nplot(imp)\n\n\n\nimp$results\n#> feature importance.05 importance importance.95 permutation.error\n#> 1 lstat 4.321882 4.553394 4.637634 4.4285295\n#> 2 rm 3.382867 3.429617 3.636302 3.3355693\n#> 3 nox 1.787123 1.827256 1.832842 1.7771480\n#> 4 crim 1.727133 1.732542 1.739232 1.6850317\n#> 5 dis 1.661721 1.674545 1.692135 1.6286255\n#> 6 ptratio 1.601922 1.636526 1.659229 1.5916490\n#> 7 indus 1.443171 1.503451 1.518952 1.4622228\n#> 8 tax 1.369524 1.386416 1.415495 1.3483973\n#> 9 age 1.370219 1.383529 1.407349 1.3455898\n#> 10 black 1.262825 1.272092 1.281437 1.2372085\n#> 11 rad 1.105651 1.119705 1.124811 1.0890002\n#> 12 zn 1.063475 1.067127 1.070519 1.0378637\n#> 13 chas 1.016268 1.017959 1.026367 0.9900441" - }, - { - "objectID": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#partial-dependence", - "href": "posts/2018-04-30-interpretable-machine-learning-iml-and-mlr/2018-04-30-interpretable-machine-learning-iml-and-mlr.html#partial-dependence", - "title": "Interpretable Machine Learning with iml and mlr", - "section": "Partial dependence", - "text": "Partial dependence\nBesides learning which features were important, we are interested in how the features influence the predicted outcome. The Partial class implements partial dependence plots and individual conditional expectation curves. Each individual line represents the predictions" - }, - { - "objectID": "posts/2017-12-14-team-rtus-wins-munichre-datathon/2017-12-14-team-rtus-wins-munichre-datathon.html", - "href": "posts/2017-12-14-team-rtus-wins-munichre-datathon/2017-12-14-team-rtus-wins-munichre-datathon.html", - "title": "Team Rtus wins Munich Re Datathon with mlr", - "section": "", - "text": "On the weekend of November 17. - 19. five brave data-knights from team “Rtus and the knights of the data.table” took on the challenge to compete in a datathon organized by Munich Re in its Munich-based innovation lab. Team Rtus was formed in April this year by a bunch of statistics students from LMU with the purpose to prove their data-skills in competitions with other teams from various backgrounds. The datathon was centered around the topic “the effects of climate change and hurricane events on modern reinsurance business” and after two days of very intensive battles with databases, web-crawlers and advanced machine learning modelling (using the famous R-library mlr), they managed to prevail against strong professional competitors and won the best overall price. After victories at the Datafest at Mannheim University and the Telefonica data challenge earlier this year, this was the last step to a hattrick for the core members of team Rtus.\nTheir way to success was the implementation of an interactive web-app that could serve as a decision support system for underwriting and tariffing units at Munich Re that are dealing with natural catastrophies. Munich Re provided the participants with databases on claims exposure in the Florida-bay, footprints of past hurricanes and tons of data on climate variable measurements over the past decades. One of the core tasks of the challenge was to define and calculate the maximum foreseeable loss and the probability of such a worst-case event to take place.\nTo answer the first question, they created a web app that calculates the expected loss of a hurricane in a certain region. To give the decision makers the opportunity to include their expert domain knowledge, they could interact with the app and set the shape and the location of the hurricane, which was modelled as a spatial Gaussian process. This is depicted in the first screenshot. Due to a NDA the true figures and descriptions in the app were altered.\n\n\n\nSuper Gauss App in normal mode.\n\n\nThe team recognised the existence of several critical nuclear power plants in this area. The shocking event of Fukushima in 2011 showed the disastrous effects that storm surges, a side effect of hurricanes, can have in combination with nuclear power plants. To account for this, team Rtus implemented the “Nuclear Power Plant” mode in the app. The function of this NPP-mode is shown in this figure:\n\n\n\nSuper Gauss App in Nuclear Power Plant mode.\n\n\nIn a next step, the team tried to provide evidence for the plausibility of such a worst-case event. The following image, based on the footprints of past hurricanes, shows that there were indeed hurricanes crossing the locations of the nuclear power plants:\n\n\n\nStorm plot to proove plausibility of the worst case.\n\n\nTo answer the second part of the question, they also created a simulation of several weather variables to forecast the probability of such heavy category 5 hurricane events. One rule of reliable statistic modelling is the inclusion of uncertainty measures in any prediction, which was integrated via the prediction intervals. Also, the user of the simulation is able to increase or decrease the estimated temperature trend that underlies the model. This screenshot illustrates the simulation app:\n\n\n\nSimulation of probability of heavy hurricanes to occur.\n\n\nThe 36 hours of intensive hacking, discussions, reiterations and fantastic team work combined with the consumption of estimated 19.68 litres of Club Mate were finally rewarded with the first place ranking and a Microsoft Surface Pro for each of the knights. “We will apply this augmentation of our weapon arsenal directly in the next data battle”, one of the knights proudly stated during the awards ceremony.\nThis portrait shows the tired but happy data knights (from left to right: Niklas Klein, Moritz Herrmann, Jann Goschenhofer, Markus Dumke and Daniel Schalk):\n\n\n\nThe succesfull team from left to right: Niklas Klein, Moritz Herrmann, Jann Goschenhofer, Markus Dumke and Daniel Schalk." - }, - { - "objectID": "posts/2018-07-25-visualize-spatial-cv/2018-07-25-visualize-spatial-cv.html", - "href": "posts/2018-07-25-visualize-spatial-cv/2018-07-25-visualize-spatial-cv.html", - "title": "Visualization of spatial cross-validation partitioning", - "section": "", - "text": "Introduction\nIn July mlr got a new feature that extended the support for spatial data: The ability to visualize spatial partitions in cross-validation (CV) 9d4f3. When one uses the resampling descriptions “SpCV” or “SpRepCV” in mlr, the k-means clustering approach after Brenning (2005) is used to partition the dataset into equally sized, spatially disjoint subsets. See also this post on r-spatial.org and the mlr vignette about spatial data for more information.\n\n\nVisualization of partitions\nWhen using random partitiong in a normal cross-validation, one is usually not interested in the random pattern of the datasets train/test split However, for spatial data this information is important since it can help identifying problematic folds (ones that did not converge or showed a bad performance) and also proves that the k-means clustering algorithm did a good job on partitioning the dataset.\nThe new function to visualize these partitions in mlr is called createSpatialResamplingPlots(). It uses ggplot2 and its new geom_sf() function to create spatial plots based on the resampling indices of a resample() object. In this post we will use the examples of the function to demonstrate its use.\nFirst, we create a resampling description rdesc using spatial partitioning with five folds and repeat it 4 times. This rdesc object is put into a resample() call together with our example task for spatial matters, spatial.task. Finally, we use the classif.qda learner to have a quick model fit.\n\nlibrary(mlr)\nrdesc = makeResampleDesc(\"SpRepCV\", folds = 5, reps = 4)\nresamp = resample(makeLearner(\"classif.qda\"), spatial.task, rdesc)\n\nNow we can use createSpatialResamplingPlots() to automatically create one plot for each fold of the resamp object. Usually we do not want to plot all repetitions of the CV. We can restrict the number of repetitions in the argument repetitions.\nBesides the required arguments task and resample, the user can specifiy the coordinate reference system that should be used for the plots. Here it is important to set the correct EPSG number in argument crs to receive accurate spatial plots. In the background, geom_sf() (more specifically coords_sf()) will transform the CRS on the fly to EPSG: 4326. This is done because lat/lon reference systems are better for plotting as UTM coordinates usually clutter the axis. However, if you insist on using UTM projection instead of WGS84 (EPSG: 4326) you can set the EPSG code of your choice in argument datum.\n\nplots = createSpatialResamplingPlots(spatial.task, resamp, crs = 32717,\n repetitions = 2, x.axis.breaks = c(-79.065, -79.085),\n y.axis.breaks = c(-3.970, -4))\n\nTo avoid overlapping axis breaks you might want to set the axis breaks on your own as we did here.\nNow we got a list of 2L back from createSpatialResamplingPlots(). In the first list we got all the plots, one for each fold. Since we used two repetitions and five folds, we have a total of ten instances in it.\nThe second list consists of labels for each plot. These are automatically created by createSpatialResamplingPlots() and can serve as titles later on. Note that for now we just created the ggplot objects (stored in the first list of the plots object). We still need to plot them!\nSingle ggplot objects can be plotted by just extracting a certain object from the list, e.g. plots[[1]][[3]]. This would plot fold #3 of repetition one.\n\nplots[[1]][[3]]\n\n\n\n\n\n\n\n\nHowever usually we want to visualize all plots in a grid. For this purpose we highly recommend to use the cowplot package and its function plot_grid().\nThe returned objects of createSpatialResamplingPlots() are already tailored to be used with this function. We just need to hand over the list of plots and (optional) the labels:\n\ncowplot::plot_grid(plotlist = plots[[\"Plots\"]], ncol = 5, nrow = 2,\n labels = plots[[\"Labels\"]])\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n\n\n\n\n\n\n\n\n\nMultiple resample objects\ncreateSpatialResamplingPlots() is quite powerful and can also take multiple resample() objects as inputs with the aim to compare both. A typical use case is to visualize the differences between spatial and non-spatial partitioning.\nTo do so, we first create two resample() objects (one using “SpRepCV”, one using “RepCV”):\n\nrdesc1 = makeResampleDesc(\"SpRepCV\", folds = 5, reps = 4)\nr1 = resample(makeLearner(\"classif.qda\"), spatial.task, rdesc1)\nrdesc2 = makeResampleDesc(\"RepCV\", folds = 5, reps = 4)\nr2 = resample(makeLearner(\"classif.qda\"), spatial.task, rdesc2)\n\nNow we can hand over both objects using a named list. This way the list names will also directly be used as a prefix in the resulting plot labels.\n\nplots = createSpatialResamplingPlots(spatial.task,\n list(\"SpRepCV\" = r1, \"RepCV\" = r2), crs = 32717, repetitions = 1,\n x.axis.breaks = c(-79.055, -79.085), y.axis.breaks = c(-3.975, -4))\n\ncowplot::plot_grid(plotlist = plots[[\"Plots\"]], ncol = 5, nrow = 2,\n labels = plots[[\"Labels\"]])\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n## \"Symbol\" could not be found for family \"\"\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\n## font \"Symbol\" could not be found for family \"\"\n\n\n\n\n\n\n\n\n\n\nReferences\nBrenning, A. (2012). Spatial cross-validation and bootstrap for the assessment of prediction rules in remote sensing: The R package sperrorest. In 2012 IEEE International Geoscience and Remote Sensing Symposium. IEEE. https://doi.org/10.1109/igarss.2012.6352393" - }, - { - "objectID": "posts/2019-12-21-mlr-award/2019-12-21-mlr-award.html", - "href": "posts/2019-12-21-mlr-award/2019-12-21-mlr-award.html", - "title": "mlr wins Open Source Machine Learning Software Award", - "section": "", - "text": "mlr receives Open Source Machine Learning Project Award\n\nWe’re extremely proud to have received the Open Source Machine Learning Project Award at ODSC West 2019. We were joined by Tensorflow, DataKind, and SHAP, which also received awards.\nThe ODSC awards recognize projects that have made sustained contributions to open data science. The mlr project has been active for more than 8 years now, being sustained by the contributions of individual volunteers. We are especially honored to have been recognized like this as the other awarded projects are coordinated and run by companies with paid staff – this speaks to the dedication of everybody who has made mlr possible.\nLars Kotthoff received the award on behalf of the entire mlr team; you can find his presentation slides giving more background on the project here. After the award talk, Martin Binder gave a tutorial on mlr3. You can find his slides and materials here.\n\nMore information on the awards can be found here." - }, - { - "objectID": "posts/2020-11-12-announcing-mlr3spatiotempcv/announcing-mlr3spatiotempcv.html", - "href": "posts/2020-11-12-announcing-mlr3spatiotempcv/announcing-mlr3spatiotempcv.html", - "title": "Announcing mlr3spatiotempcv", - "section": "", - "text": "We are happy to announce that a new extension package has joined the CRAN family of mlr3 packages. mlr3spatiotempcv was in the works for more than a year and adds spatiotemporal resampling methods to the mlr3 ecosystem.\nSuch dedicated resampling methods make it possible to retrieve biased-reduced performance estimates in cross-validation scenarios when working with spatial, temporal, or spatiotemporal datasets. mlr3spatiotempcv does not implement new methods but rather attempts to collect existing methods.\nSo far, applying such methods in both R and the mlr ecosystem was not particularly easy since they were spread across various R packages. Usually, every R package uses a slightly different syntax for the required objects and the returned results. This not only leads to an inconvenient single-use experience but is also unpractical when working in an overarching ecosystem such as mlr3.\nWe hope that with the release of this package users are now able to seamlessly work with spatiotemporal data in mlr3. Please file issues and suggestions in the issues pane of the package.\nFor a quick and rather technical introduction please see the “Get Started” vignette. For more detailed information and a detailed walk-through, see the “Spatiotemporal Analysis” section in the mlr3book.\nTo finish with something visual, a simple example which showcases the visualization capabilities of mlr3spatiotempcv for different partitioning methods (random (non-spatial) partitioning (Fig.1) vs. k-means based partitioning (spatial) (Fig. 2)):\n\n\nWarning: Ignoring unknown parameters: crs\nIgnoring unknown parameters: crs\nIgnoring unknown parameters: crs\nIgnoring unknown parameters: crs\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\n\n\n\nWarning: Ignoring unknown parameters: crs\n\n\nWarning: Ignoring unknown parameters: crs\nIgnoring unknown parameters: crs\nIgnoring unknown parameters: crs\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font\n\"Symbol\" could not be found for family \"\"\n\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"\n\nWarning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :\nfont \"Symbol\" could not be found for family \"\"" - }, - { - "objectID": "posts/2018-07-05-whyr-conference/2018-07-05-whyr-conference.html", - "href": "posts/2018-07-05-whyr-conference/2018-07-05-whyr-conference.html", - "title": "Why R Conference", - "section": "", - "text": "This July we had the great honor to present mlr and its ecosystem at the WhyR 2018 Conference in Wroclaw in Poland. You can find the slides here. We want to thank the organizers for inviting us, providing us with great food and coffee and also many thanks all participants for showing great interest in mlr,\nWe hope this could spark further improvement and new features as a result of the many interesting discussions we had with the participants. This really encouraged us to work harder, in order to create an even better software. We also hope this created interest in participating and contributing to our project, as the community really thrives on knowledge and experience of a very diverse set of people and backgrounds." - }, - { - "objectID": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html", - "href": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html", - "title": "Exploring Learner Predictions with Partial Dependence and Functional ANOVA", - "section": "", - "text": "Learners use features to make predictions but how those features are used is often not apparent. mlr can estimate the dependence of a learned function on a subset of the feature space using generatePartialDependenceData.\nPartial dependence plots reduce the potentially high dimensional function estimated by the learner, and display a marginalized version of this function in a lower dimensional space. For example suppose \\(\\mathbb{E}[Y \\ \\| \\ X = x] = f(x)\\). With \\((x, y)\\) pairs drawn independently, a learner may estimate \\(\\hat{f}\\), which, if \\(X\\) is high dimensional can be uninterpretable. Suppose we want to approximate the relationship between some column-wise subset of \\(X\\). We partition \\(X\\) into two sets, \\(X_s\\) and \\(X_c\\) such that \\(X = X_s \\cup X_c\\), where \\(X_s\\) is a subset of \\(X\\) of interest.\nThe partial dependence of \\(f\\) on \\(X_c\\) is\n\\[f_{X_s} = \\mathbb{E}_{X_c}f(X_s, X_c).\\]\nWe can use the following estimator:\n\\[\\hat{f}_{x_s} = \\frac{1}{N} \\sum_{i = 1}^N \\hat{f}(x_s, x_{ic}).\\]\nThis is described by Friedman (2001) and in Hastie, Tibsharani, and Friedman (2009).\nThe individual conditional expectation of an observation can also be estimated using the above algorithm absent the averaging, giving \\(\\hat{f}^{(i)}_{x_s}\\) as described in Goldstein, Kapelner, Bleich, and Pitkin (2014). This allows the discovery of features of \\(\\hat{f}\\) that may be obscured by an aggregated summary of \\(\\hat{f}\\).\nThe partial derivative of the partial dependence function, \\(\\frac{\\partial \\hat f_{x_s}}{\\partial x_s}\\), and the individual conditional expectation function, \\(\\frac{\\partial \\hat{f}^{(i)}_{x_s}}{\\partial x_s}\\), can also be computed. For regression and survival tasks the partial derivative of a single feature \\(x_s\\) is the gradient of the partial dependence function, and for classification tasks where the learner can output class probabilities the Jacobian. Note that if the learner produces discontinuous partial dependence (e.g., piecewise constant functions such as decision trees, ensembles of decision trees, etc.) the derivative will be 0 (where the function is not changing) or trending towards positive or negative infinity (at the discontinuities where the derivative is undefined). Plotting the partial dependence function of such learners may give the impression that the function is not discontinuous because the prediction grid is not composed of all discontinuous points in the predictor space. This results in a line interpolating that makes the function appear to be piecewise linear (where the derivative would be defined except at the boundaries of each piece).\nThe partial derivative can be informative regarding the additivity of the learned function in certain features. If \\(\\hat{f}^{(i)}_{x_s}\\) is an additive function in a feature \\(x_s\\), then its partial derivative will not depend on any other features (\\(x_c\\)) that may have been used by the learner. Variation in the estimated partial derivative indicates that there is a region of interaction between \\(x_s\\) and \\(x_c\\) in \\(\\hat{f}\\). Similarly, instead of using the mean to estimate the expected value of the function at different values of \\(x_s\\), instead computing the variance can highlight regions of interaction between \\(x_s\\) and \\(x_c\\).\nAgain, see Goldstein, Kapelner, Bleich, and Pitkin (2014) for more details and their package ICEbox for the original implementation. The algorithm works for any supervised learner with classification, regression, and survival tasks." - }, - { - "objectID": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html#partial-dependence", - "href": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html#partial-dependence", - "title": "Exploring Learner Predictions with Partial Dependence and Functional ANOVA", - "section": "Partial Dependence", - "text": "Partial Dependence\nOur implementation, following mlr’s visualization pattern, consists of the above mentioned function generatePartialDependenceData and plotPartialDependence. The former generates input (objects of class PartialDependenceData) for the latter.\nThe first step executed by generatePartialDependenceData is to generate a feature grid for every element of the character vector features passed. The data are given by the input argument, which can be a Task or a data.frame. The feature grid can be generated in several ways. A uniformly spaced grid of length gridsize (default 10) from the empirical minimum to the empirical maximum is created by default, but arguments fmin and fmax may be used to override the empirical default (the lengths of fmin and fmax must match the length of features). Alternatively the feature data can be resampled, either by using a bootstrap or by subsampling.\nResults from generatePartialDependenceData can be visualized with plotPartialDependence.\n\nlibrary(mlr)\n\nlrn.classif = makeLearner(\"classif.ksvm\", predict.type = \"prob\")\nfit.classif = train(lrn.classif, iris.task)\npd = generatePartialDependenceData(fit.classif, iris.task, \"Petal.Width\")\npd\n\nplotPartialDependence(pd, data = iris)\n\nAs noted above, \\(x_s\\) does not have to be unidimensional. If it is not, the interaction flag must be set to TRUE. Then the individual feature grids are combined using the Cartesian product, and the estimator above is applied, producing the partial dependence for every combination of unique feature values. If the interaction flag is FALSE (the default) then by default \\(x_s\\) is assumed unidimensional, and partial dependencies are generated for each feature separately. The resulting output when interaction = FALSE has a column for each feature, and NA where the feature was not used. With one feature and a regression task the output is a line plot, with a point for each point in the corresponding feature’s grid. For classification tasks there is a line for each class (except for binary classification tasks, where the negative class is automatically dropped). The data argument to plotPartialPrediction allows the training data to be input to show the empirical marginal distribution of the data.\n\npd.lst = generatePartialDependenceData(fit.classif, iris.task, c(\"Petal.Width\", \"Petal.Length\"))\nhead(pd.lst$data)\n\ntail(pd.lst$data)\n\nplotPartialDependence(pd.lst, data = iris)\n\n\npd.int = generatePartialDependenceData(fit.classif, iris.task, c(\"Petal.Width\", \"Petal.Length\"), interaction = TRUE)\npd.int\n\nplotPartialDependence(pd.int, facet = \"Petal.Length\")\n\nWhen interaction = TRUE, plotPartialDependence can either facet over one feature, showing the conditional relationship between the other feature and \\(\\hat{f}\\) in each panel, or a tile plot. The latter is, however, not possible with multiclass classification (an example of a tile plot will be shown later).\nAt each step in the estimation of \\(\\hat{f}_{x_s}\\) a set of predictions of length \\(N\\) is generated. By default the mean prediction is used. For classification where predict.type = \"prob\" this entails the mean class probabilities. However, other summaries of the predictions may be used. For regression and survival tasks the function used here must either return one number or three, and, if the latter, the numbers must be sorted lowest to highest. For classification tasks the function must return a number for each level of the target feature.\nAs noted, the fun argument can be a function which returns three numbers (sorted low to high) for a regression task. This allows further exploration of relative feature importance. If a feature is relatively important, the bounds are necessarily tighter because the feature accounts for more of the variance of the predictions, i.e., it is “used” more by the learner. More directly setting fun = var identifies regions of interaction between \\(x_s\\) and \\(x_c\\). This can also be accomplished by computing quantiles. The wider the quantile bounds, the more variation in \\(\\hat{f}\\) is due to features other than \\(x_s\\) that is shown in the plot.\n\nlrn.regr = makeLearner(\"regr.ksvm\")\nfit.regr = train(lrn.regr, bh.task)\n\npd.ci = generatePartialDependenceData(fit.regr, bh.task, \"lstat\",\n fun = function(x) quantile(x, c(.25, .5, .75)))\npd.ci\n\nplotPartialDependence(pd.ci)\n\nIn addition to bounds based on a summary of the distribution of the conditional expectation of each observation, learners which can estimate the variance of their predictions can also be used. The argument bounds is a numeric vector of length two which is added (so the first number should be negative) to the point prediction to produce a confidence interval for the partial dependence. The default is the .025 and .975 quantiles of the Gaussian distribution.\n\nfit.se = train(makeLearner(\"regr.randomForest\", predict.type = \"se\"), bh.task)\npd.se = generatePartialDependenceData(fit.se, bh.task, c(\"lstat\", \"crim\"))\nhead(pd.se$data)\n\ntail(pd.se$data)\n\nplotPartialDependence(pd.se)\n\nAs previously mentioned if the aggregation function is not used, i.e., it is the identity, then the conditional expectation of \\(\\hat{f}^{(i)}_{x_s}\\) is estimated. If individual = TRUE then generatePartialDependenceData returns \\(N\\) partial dependence estimates made at each point in the prediction grid constructed from the features.\n\npd.ind.regr = generatePartialDependenceData(fit.regr, bh.task, \"lstat\", individual = TRUE)\npd.ind.regr\n\nplotPartialDependence(pd.ind.regr)\n\nThe resulting output, particularly the element data in the returned object, has an additional column idx which gives the index of the observation to which the row pertains.\nFor classification tasks this index references both the class and the observation index.\n\npd.ind.classif = generatePartialDependenceData(fit.classif, iris.task, \"Petal.Length\", individual = TRUE)\npd.ind.classif\nplotPartialDependence(pd.ind.classif)\n\nThe plots, at least in these forms, are difficult to interpet. Individual estimates of partial dependence can also be centered by predictions made at all \\(N\\) observations for a particular point in the prediction grid created by the features. This is controlled by the argument center which is a list of the same length as the length of the features argument and contains the values of the features desired.\n\npd.ind.classif = generatePartialDependenceData(fit.classif, iris.task, \"Petal.Length\", individual = TRUE, center = list(\"Petal.Length\" = min(iris$Petal.Length)))\nplotPartialDependence(pd.ind.classif)\n\nPartial derivatives can also be computed for individual partial dependence estimates and aggregate partial dependence. This is restricted to a single feature at a time. The derivatives of individual partial dependence estimates can be useful in finding regions of interaction between the feature for which the derivative is estimated and the features excluded. Applied to the aggregated partial dependence function they are not very informative, but when applied to the individual conditional expectations, they can be used to find regions of interaction.\n\npd.regr.der.ind = generatePartialDependenceData(fit.regr, bh.task, \"lstat\", derivative = TRUE, individual = TRUE)\nhead(pd.regr.der.ind$data)\n\nplotPartialDependence(pd.regr.der.ind)\n\n\npd.classif.der.ind = generatePartialDependenceData(fit.classif, iris.task, \"Petal.Width\", derivative = TRUE, individual = TRUE)\nhead(pd.classif.der.ind$data)\n\nplotPartialDependence(pd.classif.der.ind)\n\nThis suggests that Petal.Width interacts with some other feature in the neighborhood of \\((1.5, 2)\\) for classes “virginica” and “versicolor”." - }, - { - "objectID": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html#functional-anova", - "href": "posts/2016-08-11-exploring-learner-predictions-with-partial-dependence/2016-08-11-exploring-learner-predictions-with-partial-dependence.html#functional-anova", - "title": "Exploring Learner Predictions with Partial Dependence and Functional ANOVA", - "section": "Functional ANOVA", - "text": "Functional ANOVA\nHooker (2004) proposed the decomposition of a learned function \\(\\hat{f}\\) as a sum of lower dimensional functions \\[f(\\mathbf{x}) = g_0 + \\sum_{i = 1}^p g_{i}(x_i) + \\sum_{i \\neq j} g_{ij}(x_{ij}) + \\ldots\\] where \\(p\\) is the number of features. generateFunctionalANOVAData estimates the individual \\(g\\) functions using partial dependence. When functions depend only on one feature, they are equivalent to partial dependence, but a \\(g\\) function which depends on more than one feature is the “effect” of only those features: lower dimensional “effects” are removed.\n\\[\\hat{g}_u(x) = \\frac{1}{N} \\sum_{i = 1}^N \\left( \\hat{f}(x) - \\sum_{v \\subset u} g_v(x) \\right)\\]\nHere \\(u\\) is a subset of \\({1, \\ldots, p}\\). When \\(\\|v\\| = 1\\) \\(g_v\\) can be directly computed by computing the bivariate partial dependence of \\(\\hat{f}\\) on \\(x_u\\) and then subtracting off the univariate partial dependences of the features contained in \\(v\\).\nAlthough this decomposition is generalizable to classification it is currently only available for regression tasks.\n\nlrn.regr = makeLearner(\"regr.ksvm\")\nfit.regr = train(lrn.regr, bh.task)\n\nfa = generateFunctionalANOVAData(fit.regr, bh.task, \"lstat\", depth = 1, fun = median)\nfa\n\npd.regr = generatePartialDependenceData(fit.regr, bh.task, \"lstat\", fun = median)\npd.regr\n\nThe depth argument is similar to the interaction argument in generatePartialDependenceData but instead of specifying whether all of joint “effect” of all the features is computed, it determines whether “effects” of all subsets of the features given the specified depth are computed. So, for example, with \\(p\\) features and depth 1, the univariate partial dependence is returned. If, instead, depth = 2, then all possible bivariate functional ANOVA effects are returned. This is done by computing the univariate partial dependence for each feature and subtracting it from the bivariate partial dependence for each possible pair.\n\nfa.bv = generateFunctionalANOVAData(fit.regr, bh.task, c(\"crim\", \"lstat\", \"age\"), depth = 2)\nfa.bv\n\nnames(table(fa.bv$data$effect)) ## interaction effects estimated\n\nPlotting univariate and bivariate functional ANOVA components works the same as for partial dependence.\n\nfa.bv = generateFunctionalANOVAData(fit.regr, bh.task, c(\"crim\", \"lstat\"), depth = 2)\nplotPartialDependence(fa.bv, geom = \"tile\", data = getTaskData(bh.task))\n\nWhen overplotting the training data on the plot it is easy to see that much of the variation of the effect is due to extrapolation. Although it hasn’t been implemented yet, weighting the functional ANOVA appropriately can ensure that the estimated effects do not depend (or depend less) on regions of the feature space which are sparse.\nI also plan on implementing the faster estimation algorith for expanding the functionality of the functional ANOVA function include faster computation using the algorithm in Hooker (2007) and weighting (in order to avoid excessive reliance on points of extrapolation) using outlier detection or joint density estimation." - }, - { - "objectID": "posts/2020-03-06-mlr3-tutorial-user2020/2020-03-06-mlr3-tutorial-user2020.html", - "href": "posts/2020-03-06-mlr3-tutorial-user2020/2020-03-06-mlr3-tutorial-user2020.html", - "title": "mlr3 tutorial on useR!2020muc", - "section": "", - "text": "mlr3 tutorial at the useR!2020 European hub\nWe are thrilled that we got accepted for a tutorial at the useR!2020 satellite event in Munich on July 7th. Bernd Bischl and Michel Lang will give an introduction to mlr3, the successor of the mlr package for machine learning in R.\nThe main objective of the tutorial is to introduce and familiarize users with mlr3 and its ecosystem of extension packages. This will allow participants to take advantage of its functionality for their own projects, in particular:\n\nhow to benchmark and compare different machine learning approaches in a statistically sound manner,\nhow to build complex machine learning workflows with mlr3pipelines, including preprocessing and stacked ensembles,\nautomatic hyperparameter tuning and pipeline optimization (AutoML) with mlr3tuning,\nhow to get the technical “nitty-gritties” for machine learning experiments right, e.g., speed up by parallelization, encapsulation of experiments in external processes or working on databases.\n\nAfter the tutorial, participants will be able to implement complex solutions to real-world machine learning problems, and to evaluate the different design decisions necessary for such implementations, in particular the choice between different modelling and preprocessing techniques.\n\n\nAbout the useR!2020 satellite event in Munich\nThe conference is a satellite event of the “official” useR!2020 in St. Louis, USA. It is actively supported by the R Foundation, and there will be streamed keynote talks from the US to Europe and vice versa. The following speakers will give a keynote in Munich (you can find the tentative programme here):\n\nKurt Hornik and Uwe Ligges about the future of CRAN\nAnna Krystalli about computational reproducibility\nPrzemysław Biecek about explorable and explainable machine learning models\n\nNote that there is still time to submit an abstract for a talk, a lightning talk or a poster. Registration is also open now!" - }, - { - "objectID": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html", - "href": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html", - "title": "mlr loves OpenML", - "section": "", - "text": "OpenML stands for Open Machine Learning and is an online platform, which aims at supporting collaborative machine learning online. It is an Open Science project that allows its users to share data, code and machine learning experiments.\nAt the time of writing this blog I am in Eindoven at an OpenML workshop, where developers and scientists meet to work on improving the project. Some of these people are R users and they (we) are developing an R package that communicates with the OpenML platform." - }, - { - "objectID": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#openml-in-r", - "href": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#openml-in-r", - "title": "mlr loves OpenML", - "section": "OpenML in R", - "text": "OpenML in R\nThe OpenML R package can list and download data sets and machine learning tasks (prediction challenges). In R one can run algorithms on the these data sets/tasks and then upload the results to OpenML. After successful uploading, the website shows how well the algorithm performs. To run the algorithm on a given task the OpenML R package builds on the mlr package. mlr understands what a task is and can run learners on that task. So all the OpenML package needs to do is convert the OpenML objects to objects mlr understands and then mlr deals with the learning." - }, - { - "objectID": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#a-small-case-study", - "href": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#a-small-case-study", - "title": "mlr loves OpenML", - "section": "A small case study", - "text": "A small case study\nWe want to create a little study on the OpenML website, in which we compare different types of Support Vector Machines. The study gets an ID assigned to it, which in our case is 27. We use the function ksvm (with different settings of the function argument type) from package kernlab, which is integrated in mlr (“classif.ksvm”).\n\nFor details on installing and setting up the OpenML R package please see the guide on GitHub.\nLet’s start conducting the study:\n\nLoad the packages and list all tasks which have between 100 and 500 observations.\n\n\nlibrary(\"OpenML\")\nlibrary(\"mlr\")\nlibrary(\"farff\")\nlibrary(\"BBmisc\")\n\n\ndsize = c(100, 500)\ntaskinfo_all = listOMLTasks(number.of.instances = dsize)\n\n\nSelect all supervised classification tasks that do 10-fold cross-validation and choose only one task per data set. To keep the study simple and fast to compute, select only the first three tasks.\n\n\ntaskinfo_10cv = subset(taskinfo_all, task.type == \"Supervised Classification\" &\n estimation.procedure == \"10-fold Crossvalidation\" &\n evaluation.measures == \"predictive_accuracy\" &\n number.of.missing.values == 0 &\n number.of.classes %in% c(2, 4))\ntaskinfo = taskinfo_10cv[1:3, ]\n\n\nCreate the learners we want to compare.\n\n\nlrn.list = list(\n makeLearner(\"classif.ksvm\", type = \"C-svc\"),\n makeLearner(\"classif.ksvm\", type = \"kbb-svc\"),\n makeLearner(\"classif.ksvm\", type = \"spoc-svc\")\n)\n\n\nRun the learners on the three tasks.\n\n\ngrid = expand.grid(task.id = taskinfo$task.id,\n lrn.ind = seq_along(lrn.list))\n\nruns = lapply(seq_row(grid), function(i) {\n message(i)\n task = getOMLTask(grid$task.id[i])\n ind = grid$lrn.ind[i]\n runTaskMlr(task, lrn.list[[ind]])\n})\n\n\nAnd finally upload the runs to OpenML. The upload function (uploadOMLRun) returns the ID of the uploaded run object. When uploading runs that are part of a certain study, tag it with study_ and the study ID. After uploading the runs appear on the website and can be found using the tag or via the study homepage.\n\n\n## please do not spam the OpenML server by uploading these\n## tasks. I already did that.\nrun.id = lapply(runs, uploadOMLRun, tags = \"study_27\")\n\nTo show the results of our study, list the run evaluations and make a nice plot.\n\nevals = listOMLRunEvaluations(tag = \"study_27\")\n\nevals$task.id = as.factor(evals$task.id)\nevals$setup.id = as.factor(evals$setup.id)\n\nlibrary(\"ggplot2\")\nggplot(evals, aes(x = setup.id, y = predictive.accuracy,\n color = data.name, group = task.id)) +\n geom_point() + geom_line()\n\nNow you can go ahead and create a bigger study using the techniques you have learned." - }, - { - "objectID": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#further-infos", - "href": "posts/2016-09-09-mlr-loves-openml/2016-09-09-mlr-loves-openml.html#further-infos", - "title": "mlr loves OpenML", - "section": "Further infos", - "text": "Further infos\nIf you are interested in more, check out the OpenML blog, the paper and the GitHub repos." - }, - { - "objectID": "posts/2021-12-01-announcing-mlr3spatial/announcing-mlr3spatial.html", - "href": "posts/2021-12-01-announcing-mlr3spatial/announcing-mlr3spatial.html", - "title": "Announcing mlr3spatial", - "section": "", - "text": "s We are happy to announce that mlr3spatial has been released on CRAN in November 2021. mlr3spatial simplifies the handling of spatial objects in the mlr3 ecosystem. Before mlr3spatial, the user had to extract tabular data from spatial objects to train a model or predict spatial data.\nNow, mlr3 Tasks can directly read from spatial objects via specialized Data Backends. Such tasks can be used to train a model or to perform resampling just like any other mlr3 task. We support spatial raster objects created by the terra, raster and stars packages with DataBackendRaster. Additionally, vector data created with the sf package is handled with DataBackendVector.\nThe predict_raster() function creates spatial rasters and features with predictions from mlr3 learners. We only have to pass a task with a spatial data backend which provides the data and spatial reference. To avoid memory issues with large raster files, prediction is done in chunks. For this, the raster map is divided into multiple horizontal strips. The vertical extension of these strips is controlled by the chunksize parameter. The actual memory usage per core is a multiple of the specified chunk size. We choose a default chunk size of 200 Megabytes which should work on most consumer grade machines. If more memory is available, a larger chunk size accelerates the prediction process.\nOne after the other, the raster chunks are loaded into memory and the prediction is written to disk. Finally, the complete raster is available on disk. The learner can also make use of future-based parallelization to accelerate the predictions. The vignette on “Benchmarking parallel predictions” showcases the parallelization capabilities of mlr3spatial." - }, - { - "objectID": "posts/2021-12-01-announcing-mlr3spatial/announcing-mlr3spatial.html#use-case---landsat7-data-as-stars-object", - "href": "posts/2021-12-01-announcing-mlr3spatial/announcing-mlr3spatial.html#use-case---landsat7-data-as-stars-object", - "title": "Announcing mlr3spatial", - "section": "Use Case - Landsat7 data as {stars} object", - "text": "Use Case - Landsat7 data as {stars} object\n\nData Preparation\n\n\n\nFirst, the TIFF files is read via stars::read_stars() and put into a DataBackendRaster. The DataBackend is then used to create a regression task with the response being layer.1.\n\n\n (122848 x 6)\n* Target: band1\n* Properties: -\n* Features (5):\n - dbl (5): band2, band3, band4, band5, band6\n\n\nFor large raster files with millions of values it helps to predict in parallel. To enable this, set learner$parallel_predict = TRUE and initiate a parallel plan via {future}, e.g. via plan(\"multisession\"). Since this is only an example, parallelization is not enabled here. Here we will use a simple regression tree as an example learner. In practice you might want to use a different learner - you can find an overview of available learners here.\n\n\n: Regression Tree\n* Model: rpart\n* Parameters: xval=0\n* Packages: mlr3, rpart\n* Predict Types: [response]\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, selected_features, weights\n\n\n\n\nPrediction\nFor prediction predict_spatial() is used. It will return a raster file which contains the predictions. Users can select which R spatial format the returned raster should have. Note that by default mlr3spatial operates with SpatRaster objects from package terra. If a different output format is requested (e.g. \"stars\"), coercion is happening in the background which might take some time.\n\n\nINFO [18:06:17.232] [mlr3] Start raster prediction \nINFO [18:06:17.266] [mlr3] Prediction is executed with a chunksize of 200 Megabytes, 1 chunk(s) in total, 122848 values per chunk \nINFO [18:06:17.495] [mlr3] Chunk 1 of 1 finished \nINFO [18:06:17.504] [mlr3] Finished raster prediction in 0 seconds \n\n\nstars object with 2 dimensions and 1 attribute\nattribute(s):\n Min. 1st Qu. Median Mean 3rd Qu. Max.\ncadmium 62.3629 70.30233 77.01695 79.05135 89.2809 118.1429\ndimension(s):\n from to offset delta refsys point values x/y\nx 1 349 288776 28.5 SIRGAS 2000 / UTM zone 25S FALSE NULL [x]\ny 1 352 9120761 -28.5 SIRGAS 2000 / UTM zone 25S FALSE NULL [y]\n\n\n\n\nVisualization\nFinally we can plot the predictions. The color vector is extracted from the “viridis” color palette via dput(viridis::viridis_pal()(5)) and provided to the S3 plot() call, which makes use of the S3 plot method within the stars package in this scenario." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html", - "title": "mlr Workshop 2021 Recap", - "section": "", - "text": "This blog post is a recap of the mlr-org workshop 2021 which took place from the 28th of September until the 10th of October.\nFirst of all, we would like to thank all people and organizations which made this workshop possible:\nIf you like our work and want to see more of such workshops where we are able to devot our full time to mlr3, consider becoming a GitHub sponsor ❤️" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3", - "text": "mlr3\nFor our core package {mlr3} we mainly focused on maintenance and issue curation. A little new goodie relates to scoring and aggregating: one can now conduct more complex benchmarks using different evaluation metics." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3pipelines", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3pipelines", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3pipelines", - "text": "mlr3pipelines\n{mlr3pipelines} has also seen a lot of maintenance in the first place. In addition, we did some code profiling and were able to improve the performance a bit by reducing the overhead when cloning Paramsets and learners. This comes with the new %>>!% operator which concatenetes partial Graphs in place and is essentially carrying the memory and speed improvements from above.\nA new sugar function pos() now exists, making it possible to initialize multiple pipe operators more easily.\nWe also started working on adding more sugar to {paradox} (e.g. trafos) which should make life easier in {mlr3pipelines} in the long run.\nLast, we overwhauled some error messages internally to make them more descriptive for users." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#tuning", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#tuning", - "title": "mlr Workshop 2021 Recap", - "section": "Tuning", - "text": "Tuning\nmlr3tuning / bbotk / mlr3mbo / mlr3hyperband" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3hyperband", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3hyperband", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3hyperband", - "text": "mlr3hyperband\nOne focus with respect to tuning was on “hot starting” learners. This means that learners can save their tuning state at an arbitrary point and can be retrained later on starting from this point. This means that tuning can be done in decoupled chunks instead of one big task. This is especially helpful for certain tuning methods like “hyperband”, which is why we have listed this feature in this section.\nProposal" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#bbotk", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#bbotk", - "title": "mlr Workshop 2021 Recap", - "section": "bbotk", - "text": "bbotk\nAnother improvement for {bbotk} and all tuning packages was the support for asynchronous evaluations of hyperparameter configurations. Before, one had to wait until all hyperband configurations were evaluated to propose a new point. Now, new points can be evaluated at any time, making more efficient use of parallelization. This also avoids the issue of waiting on a few slow workers with an ineffecient configuriation that takes a long time to finish." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3mbo", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3mbo", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3mbo", - "text": "mlr3mbo\nWe made great progress in finalizing {mlr3mbo}. {mlr3mbo} is a flexible Bayesian optimization toolkit and much more modular than its predecessor {mlrMBO}. Most of the functionality {mlrMBO} offers is already integrated, and we are looking forward to a CRAN release in the near future - in the meantime make sure to check out the package on GitHub." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3learners", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3learners", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3learners", - "text": "mlr3learners\nFor our curated set of learners we updated some ParamSet to the most recent versions and ensured that our custom “Paramtest”, i.e. the heuristic how we validate the ParamSets of mlr3learners against their upstream packages, works again smoothly." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3fairness", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3fairness", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3fairness", - "text": "mlr3fairness\n{mlr3} now allows for bias auditing and debiasing through mlr3fairness for arbitrary learners. The package contains Fairness Metrics, Debiasing Strategies and Visualizations for arbitrary models trained through {mlr3}. We plan a to realease on CRAN soon, but the package is already usable. If you wanna learn more, check out the tutorials for the individual components:\n\nFairness Metrics\nDebiasing Methods\nFairness Visualizations\n\n\n\n\nmlr3fairness example plots" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3viz", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3viz", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3viz", - "text": "mlr3viz\nWe created a new ggplot2 theme theme_mlr3() which is being applied by default to all plots created with {mlr3viz}, i.e. all autoplot() methods in mlr3. This theme is heavily inspired by ggpubr::theme_pubr(). In addition to the theme, all mlr3viz plots now use the viridis color palette by default. Last, we have added some information how users can easily apply theming changes to the plots returned by autoplot().\n\n\n\ntheme_mlr3() example plot" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3spatial", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3spatial", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3spatial", - "text": "mlr3spatial\nmlr3spatial is a new package for spatial backends able to handle {sf}, {stars} and {terra} objects. It is capable of performing predictions on spatial raster objects in parallel using the internal mlr3 parallelization heuristic (which uses the {future} framework under the hood). Together with the {mlr3spatiotempcv} package it extends the support for spatial machine learning in mlr3." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr-org-meta", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr-org-meta", - "title": "mlr Workshop 2021 Recap", - "section": "mlr-org meta", - "text": "mlr-org meta" - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#roadmap", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#roadmap", - "title": "mlr Workshop 2021 Recap", - "section": "Roadmap", - "text": "Roadmap\nWe have createad a Roadmap of upcoming packages and planned features across the whole mlr3 ecosystem. This roadmap aims both for internal organization and giving external people an idea of what is upcoming. The roadmap is quite new and not yet fully operational at the time this blog post got released - however, we encourage to look at it from time to time as we try to keep it up-to-date." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3-wiki", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3-wiki", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3 wiki", - "text": "mlr3 wiki\nWe have updated and reorganized our mlr3 wiki. It now has a better sidebar appearance for easier navigation and we have updated the sections with respect to their content." - }, - { - "objectID": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3book", - "href": "posts/2021-10-07-mlr-workshop-2021-recap/mlr-workshop-2021-recap.html#mlr3book", - "title": "mlr Workshop 2021 Recap", - "section": "mlr3book", - "text": "mlr3book\n\nText and structure improvements (mainly for the basics chapter)\nNew introduction" - }, - { - "objectID": "posts/2017-03-23-new-mlr-logo/2017-03-23-new-mlr-logo.html", - "href": "posts/2017-03-23-new-mlr-logo/2017-03-23-new-mlr-logo.html", - "title": "New mlr Logo", - "section": "", - "text": "We at mlr are currently deciding on a new logo, and in the spirit of open-source, we would like to involve the community in the voting process!\nYou can vote for your favorite logo on GitHub by reacting to the logo with a +1.\nThanks to Hannah Atkin for designing the logos! ing the logos!" - }, - { - "objectID": "posts/2017-02-13-mlr-gsoc/2017-02-13-mlr-gsoc.html", - "href": "posts/2017-02-13-mlr-gsoc/2017-02-13-mlr-gsoc.html", - "title": "GSOC", - "section": "", - "text": "We are happy to announce that we applied for a another Google Summer of Code project in 2017.\nOperator Based Machine Learning Pipeline Construction\nWe aim to change the way we are currently doing data preprocessing in mlr. Have a look at the proposal linked above for more details.\nIf you are interested in doing this project, have a look at the tests and requirements." - }, - { - "objectID": "posts/2018-01-10-stepwise-bayesian-optimization-with-mlrmbo/2018-01-10-stepwise-bayesian-optimization-with-mlrmbo.html", - "href": "posts/2018-01-10-stepwise-bayesian-optimization-with-mlrmbo/2018-01-10-stepwise-bayesian-optimization-with-mlrmbo.html", - "title": "Stepwise Bayesian Optimization with mlrMBO", - "section": "", - "text": "With the release of the new version of mlrMBO we added some minor fixes and added a practical feature called Human-in-the-loop MBO. It enables you to sequentially\n\nvisualize the state of the surrogate model,\nobtain the suggested parameter configuration for the next iteration and\nupdate the surrogate model with arbitrary evaluations.\n\nIn the following we will demonstrate this feature on a simple example.\nFirst we need an objective function we want to optimize. For this post a simple function will suffice but note that this function could also be an external process as in this mode mlrMBO does not need to access the objective function as you will only have to pass the results of the function to mlrMBO.\n\nlibrary(mlrMBO)\nlibrary(ggplot2)\nset.seed(1)\n\nfun = function(x) {\n x^2 + sin(2 * pi * x) * cos(0.3 * pi * x)\n}\n\nHowever we still need to define the our search space. In this case we look for a real valued value between -3 and 3. For more hints about how to define ParamSets you can look here or in the help of ParamHelpers.\n\nps = makeParamSet(\n makeNumericParam(\"x\", lower = -3, upper = 3)\n)\n\nWe also need some initial evaluations to start the optimization. The design has to be passed as a data.frame with one column for each dimension of the search space and one column y for the outcomes of the objective function.\n\ndes = generateDesign(n = 3, par.set = ps)\ndes$y = apply(des, 1, fun)\ndes\n## x y\n## 1 -1.1835844 0.9988801\n## 2 -0.5966361 0.8386779\n## 3 2.7967794 8.6592973\n\nWith these values we can initialize our sequential MBO object.\n\nctrl = makeMBOControl()\nctrl = setMBOControlInfill(ctrl, crit = crit.ei)\nopt.state = initSMBO(\n par.set = ps,\n design = des,\n control = ctrl,\n minimize = TRUE,\n noisy = FALSE)\n\nThe opt.state now contains all necessary information for the optimization. We can even plot it to see how the Gaussian process models the objective function.\n\nplot(opt.state)\n\n\n\n\nIn the first panel the expected improvement (\\(EI = E(y_{min}-\\hat{y})\\)) (see Jones et.al.) is plotted over the search space. The maximum of the EI indicates the point that we should evaluate next. The second panel shows the mean prediction of the surrogate model, which is the Gaussian regression model aka Kriging in this example. The third panel shows the uncertainty prediction of the surrogate. We can see, that the EI is high at points, where the mean prediction is low and/or the uncertainty is high.\nTo obtain the specific configuration suggested by mlrMBO for the next evaluation of the objective we can run:\n\nprop = proposePoints(opt.state)\nprop\n## $prop.points\n## x\n## 798 -2.999998\n## \n## $propose.time\n## [1] 0.068\n## \n## $prop.type\n## [1] \"infill_ei\"\n## \n## $crit.vals\n## [,1]\n## [1,] -0.3733869\n## \n## $crit.components\n## se mean\n## 1 2.889951 3.031346\n## \n## $errors.model\n## [1] NA\n## \n## attr(,\"class\")\n## [1] \"Proposal\" \"list\"\n\nWe will execute our objective function with the suggested value for x and feed it back to mlrMBO:\n\ny = fun(prop$prop.points$x)\ny\n## [1] 8.999972\nupdateSMBO(opt.state, x = prop$prop.points, y = y)\n\nThe nice thing about the human-in-the-loop mode is, that you don’t have to stick to the suggestion. In other words we can feed the model with values without receiving a proposal. Let’s assume we have an expert who tells us to evaluate the values \\(x=-1\\) and \\(x=1\\) we can easily do so:\n\ncustom.prop = data.frame(x = c(-1,1))\nys = apply(custom.prop, 1, fun)\nupdateSMBO(opt.state, x = custom.prop, y = as.list(ys))\nplot(opt.state, scale.panels = TRUE)\n## Warning: Removed 1 rows containing missing values (geom_point).\n\n\n\n\nWe can also automate the process easily:\n\nreplicate(3, {\n prop = proposePoints(opt.state)\n y = fun(prop$prop.points$x)\n updateSMBO(opt.state, x = prop$prop.points, y = y)\n})\n\nNote: We suggest to use the normal mlrMBO if you are only doing this as mlrMBO has more advanced logging, termination and handling of errors etc.\nLet’s see how the surrogate models the true objective function after having seen seven configurations:\n\nplot(opt.state, scale.panels = TRUE)\n## Warning: Removed 1 rows containing missing values (geom_point).\n\n\n\n\nYou can convert the opt.state object from this run to a normal mlrMBO result object like this:\n\nres = finalizeSMBO(opt.state)\nres\n## Recommended parameters:\n## x=-0.22\n## Objective: y = -0.913\n## \n## Optimization path\n## 3 + 6 entries in total, displaying last 10 (or less):\n## x y dob eol error.message exec.time ei error.model\n## 1 -1.1835844 0.9988801 0 NA NA NA \n## 2 -0.5966361 0.8386779 0 NA NA NA \n## 3 2.7967794 8.6592973 0 NA NA NA \n## 4 -2.9999977 8.9999724 1 NA NA -0.3733869 \n## 5 -1.0000000 1.0000000 2 NA NA -0.3136183 \n## 6 1.0000000 1.0000000 2 NA NA -0.1366681 \n## 7 0.3009402 1.0018545 3 NA NA -0.7751006 \n## 8 -0.2198973 -0.9127938 4 NA NA -0.1568373 \n## 9 -2.2260548 5.4527615 5 NA NA -0.2965027 \n## train.time prop.type propose.time se mean\n## 1 NA initdesign NA NA NA\n## 2 NA initdesign NA NA NA\n## 3 NA initdesign NA NA NA\n## 4 0 manual NA 2.8899512 3.0313463\n## 5 0 manual NA 0.5709681 0.6836897\n## 6 NA NA 3.3578427 5.3792305\n## 7 0 manual NA 1.2338757 0.3493772\n## 8 0 manual NA 0.4512026 0.8870804\n## 9 0 manual NA 3.5929994 2.6982361\n\nNote: You can always run the human-in-the-loop MBO on res$final.opt.state.\nFor the curious, let’s see how our original function actually looks like and which points we evaluated during our optimization:\n\nplot(fun, -3, 3)\npoints(x = getOptPathX(res$opt.path)$x, y = getOptPathY(res$opt.path))\n\n\n\n\nWe can see, that we got pretty close to the global optimum and that the surrogate in the previous plot models the objective quite accurate.\nFor more in-depth information look at the Vignette for Human-in-the-loop MBO and check out the other topics of our mlrMBO page." - }, - { - "objectID": "posts/2020-08-04-user-2020-tutorial-on-mlr3-mlr3tuning-and-pipelines/user-2020-tutorial-on-mlr3-mlr3tuning-and-pipelines.html", - "href": "posts/2020-08-04-user-2020-tutorial-on-mlr3-mlr3tuning-and-pipelines/user-2020-tutorial-on-mlr3-mlr3tuning-and-pipelines.html", - "title": "useR 2020 Tutorial on mlr3, mlr3tuning and mlr3pipelines", - "section": "", - "text": "We would like to invite you to our useR 2020 tutorial on mlr3, mlr3tuning and mlr3pipelines taught by Bernd Bischl and Michel Lang. The tutorial will take place on 7th August at 10:00 (UTC-5). You can find more information and the registration link on meetup.com." - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html", - "title": "mlr 2.10", - "section": "", - "text": "mlr 2.10 is now on CRAN. Please update your package if you haven’t done so in a while.\nHere is an overview of the changes:" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---general", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---general", - "title": "mlr 2.10", - "section": "functions - general", - "text": "functions - general\n\nfixed bug in resample when using predict = “train” (issue #1284)\nupdate to irace 2.0 – there are algorithmic changes in irace that may affect performance\ngenerateFilterValuesData: fixed a bug wrt feature ordering\nimputeLearner: fixed a bug when data actually contained no NAs\nprint.Learner: if a learner hyperpar was set to value “NA” this was not displayed in printer\nmakeLearner, setHyperPars: if you mistype a learner or hyperpar name, mlr uses fuzzy matching to suggest the 3 closest names in the message\ntuneParams: tuning with irace is now also parallelized, i.e., different learner configs are evaluated in parallel.\nbenchmark: mini fix, arg ‘learners’ now also accepts class strings\nobject printers: some mlr printers show head previews of data.frames. these now also print info on the total nr of rows and cols and are less confusing\naggregations: have better properties now, they know whether they require training or test set evals\nthe filter methods have better R docs\nfilter randomForestSRC.var.select: new arg “method”\nfilter mrmr: fixed some smaller bugs and updated properties\ngenerateLearningCurveData: also accepts single learner, does not require a list\nplotThreshVsPerf: added “measures” arg\nplotPartialDependence: can create tile plots with joint partial dependence on two features for multiclass classification by facetting across the classes\ngeneratePartialDependenceData and generateFunctionalANOVAData: expanded “fun” argument to allow for calculation of weights\nnew “?mlrFamilies” manual page which lists all families and the functions belonging to it\nwe are converging on data.table as a standard internally, this should not change any API behavior on the outside, though\ngenerateHyperParsEffectData and plotHyperParsEffect now support more than 2 hyperparameters\nlinear.correlation, rank.correlation, anova.test: use Rfast instead of FSelector/custom implementation now, performance should be much better\nuse of our own colAUC function instead of the ROCR package for AUC calculation to improve performance\nwe output resample performance messages for every iteration now\nperformance improvements for the auc measure\ncreateDummyFeatures supports vectors now\nremoved the pretty.names argument from plotHyperParsEffect – labels can be set though normal ggplot2 functions on the returned object\nFixed a bad bug in resample, the slot “runtime” or a ResampleResult, when the runtime was measured not in seconds but e.g. mins. R measures then potentially in mins, but mlr claimed it would be seconds.\nNew “dummy” learners (that disregard features completely) can be fitted now for baseline comparisons, see “featureless” learners below." - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---new", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---new", - "title": "mlr 2.10", - "section": "functions - new", - "text": "functions - new\n\nfilter: randomForest.importance\ngenerateFeatureImportanceData: permutation-based feature importance and local importance\ngetFeatureImportanceLearner: new Learner API function\ngetFeatureImportance: top level function to extract feature importance information\ncalculateROCMeasures\ncalculateConfusionMatrix: new confusion-matrix like function that calculates and tables many receiver operator measures\nmakeLearners: create multiple learners at once\ngetLearnerId, getLearnerType, getLearnerPredictType, getLearnerPackages\ngetLearnerParamSet, getLearnerParVals\ngetRRPredictionList\naddRRMeasure\nplotResiduals\ngetLearnerShortName\nmergeBenchmarkResults" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---renamed", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---renamed", - "title": "mlr 2.10", - "section": "functions - renamed", - "text": "functions - renamed\n\nRenamed rf.importance filter (now deprecated) to randomForestSRC.var.rfsrc\nRenamed rf.min.depth filter (now deprecated) to randomForestSRC.var.select\nRenamed getConfMatrix (now deprecated) to calculateConfusionMatrix\nRenamed setId (now deprecated) to setLearnerId" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---removed", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#functions---removed", - "title": "mlr 2.10", - "section": "functions - removed", - "text": "functions - removed\n\nmergeBenchmarkResultLearner, mergeBenchmarkResultTask" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---general", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---general", - "title": "mlr 2.10", - "section": "learners - general", - "text": "learners - general\n\nclassif.ada: fixed some param problem with rpart.control params\nclassif.cforest, regr.cforest, surv.cforest: removed parameters “minprob”, “pvalue”, “randomsplits” as these are set internally and cannot be changed by the user\nregr.GPfit: some more params for correlation kernel\nclassif.xgboost, regr.xgboost: can now properly handle NAs (property was missing and other problems), added “colsample_bylevel” parameter\nadapted {classif,regr,surv}.ranger parameters for new ranger version" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---new", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---new", - "title": "mlr 2.10", - "section": "learners - new", - "text": "learners - new\n\nmultilabel.cforest\nsurv.gbm\nregr.cvglmnet\n{classif,regr,surv}.gamboost\nclassif.earth\n{classif,regr}.evtree\n{classif,regr}.evtree" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---removed", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#learners---removed", - "title": "mlr 2.10", - "section": "learners - removed", - "text": "learners - removed\n\nclassif.randomForestSRCSyn, regr.randomForestSRCSyn: due to continued stability issues" - }, - { - "objectID": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#measures---new", - "href": "posts/2017-02-13-mlr-210/2017-02-13-mlr-210.html#measures---new", - "title": "mlr 2.10", - "section": "measures - new", - "text": "measures - new\n\nssr, qsr, lsr\nrrse, rae, mape\nkappa, wkappa\nmsle, rmsle" - }, - { - "objectID": "posts/2019-07-29-mlr3-0-1-0/2019-07-29-mlr3-0-1-0.html", - "href": "posts/2019-07-29-mlr3-0-1-0/2019-07-29-mlr3-0-1-0.html", - "title": "mlr3-0.1.0", - "section": "", - "text": "mlr3 - Initial release\nThe mlr-org team is very proud to present the initial release of the mlr3 machine-learning framework for R.\nmlr3 comes with a clean object-oriented-design using the R6 class system. With this, it overcomes the limitations of R’s S3 classes. It is a rewrite of the well-known mlr package which provides a convenient way of accessing many algorithms in R through a consistent interface.\nWhile mlr was one big package that included everything starting from preprocessing, tuning, feature-selection to visualization, mlr3 is a package framework consisting of many packages containing specific parts of the functionality required for a complete machine-learning workflow.\n\n\nBackground - why a rewrite?\nThe addition of many features to mlr has led to a “feature creep” which makes it hard to maintain and extend.\nDue to the many tests in mlr, a full CI run of the package on Travis CI takes more than 30 minutes. This does not include the installation of dependencies, running R CMD check or building the pkgdown site (which includes more than 20 vignettes with executable code). The dependency installation alone takes 1 hour(!). This is due to the huge number of packages that mlr imports to be able to use all the algorithms for which it provides a unified interface. On a vanilla CI build without cache there would be 326(!) packages to be installed.\nmlr consists of roughly 40k lines of R code.\ngithub.com/AlDanial/cloc v 1.82 T=0.98 s (1335.1 files/s, 226920.4 lines/s)\n-------------------------------------------------------------------------------\nLanguage files blank comment code\n-------------------------------------------------------------------------------\nHTML 346 16592 6276 125687\nR 878 5909 12566 40696\nRmd 44 2390 5296 2500\nMarkdown 11 196 0 1136\nXML 1 0 0 963\nYAML 4 14 5 244\nCSS 2 69 66 203\nC 3 16 33 106\nJSON 5 0 0 98\nJavaScript 2 21 9 80\nBourne Shell 4 8 6 69\nC/C++ Header 1 7 0 20\nSVG 1 0 1 11\n-------------------------------------------------------------------------------\nSUM: 1302 25222 24258 171813\n-------------------------------------------------------------------------------\nAdding any feature has become a huge pain since it requires passing down objects through many functions. In addition, a change to one part of the project might trigger unwanted side-effects at other places or even break functions. Even though most of the package is covered by tests, this creates a high wall of reluctance to extend the p" - }, - { - "objectID": "posts/2017-09-01-openml-workshop-2017/2017-09-01-openml-workshop-2017.html", - "href": "posts/2017-09-01-openml-workshop-2017/2017-09-01-openml-workshop-2017.html", - "title": "OpenML Workshop 2017", - "section": "", - "text": "The field of Machine Learning has grown tremendously over the last years, and is a key component of data-driven science. Data analysis algorithms are being invented and used every day, but their results and experiments are published almost exclusively in journals or separated repositories. However, data by itself has no value. It’s the ever-changing ecosystem surrounding data that gives it meaning.\nOpenML is a networked science platform that aims to connect and organize all this knowledge online, linking data, algorithms, results and people into a coherent whole so that scientists and practitioners can easy build on prior work and collaborate in real time online.\nOpenML has an online interface on openml.org, and is integrated in the most popular machine learning tools and statistical environments such as R, Python, WEKA, MOA and RapidMiner. This allows researchers and students to easily import and export data from these tools and share them with others online, fully integrated into the context of the state of the art. On OpenML, researchers can connect to each other, start projects, and build on the results of others. It automatically keeps track of how often shared work is reused so that researchers can follow the wider impact of their work and become more visible." - }, - { - "objectID": "posts/2017-09-01-openml-workshop-2017/2017-09-01-openml-workshop-2017.html#when-and-where", - "href": "posts/2017-09-01-openml-workshop-2017/2017-09-01-openml-workshop-2017.html#when-and-where", - "title": "OpenML Workshop 2017", - "section": "When and Where?", - "text": "When and Where?\nThe OpenML workshop is organized as a hackathon, an event where participants from many scientific domains present their goals and ideas, and then work on them in small teams for many hours or days at a time. Participants bring their laptops, learn how to use OpenML in tutorials, and build upon that to create something great to push their research forward. The complete OpenML development team will be available to get them started, answer questions, and implement new features on the fly.\nThe next OpenML Workshop will run from 9 October to 13 Oktober 2017, see also here." - }, - { - "objectID": "posts/2016-08-21-exploring-and-understanding-hyperparameter-tuning/2016-08-21-exploring-and-understanding-hyperparameter-tuning.html", - "href": "posts/2016-08-21-exploring-and-understanding-hyperparameter-tuning/2016-08-21-exploring-and-understanding-hyperparameter-tuning.html", - "title": "Exploring and Understanding Hyperparameter Tuning", - "section": "", - "text": "Learners use hyperparameters to achieve better performance on particular datasets. When we use a machine learning package to choose the best hyperparmeters, the relationship between changing the hyperparameter and performance might not be obvious. mlr provides several new implementations to better understand what happens when we tune hyperparameters and to help us optimize our choice of hyperparameters.\n\nBackground\nLet’s say you have a dataset, and you’re getting ready to flex your machine learning muscles. Maybe you want to do classification, or regression, or clustering. You get your dataset together and pick a few learners to evaluate.\nThe majority of learners that you might use for any of these tasks have hyperparameters that the user must tune. Hyperparameters may be able to take on a lot of possible values, so it’s typically left to the user to specify the values. If you’re using a popular machine learning library like sci-kit learn, the library will take care of this for you via cross-validation: auto-generating the optimal values for your hyperparameters. We’ll then take these best-performing hyperparameters and use those values for our learner. Essentially, we treat the optimization of hyperparameters as a black box.\nIn mlr, we want to open up that black box, so that you can make better decisions. Using the functionality built-in, we can answer questions like:\n\nHow does varying the value of a hyperparameter change the performance of the machine learning algorithm?\nOn a related note: where’s an ideal range to search for optimal hyperparameters?\nHow did the optimization algorithm (prematurely) converge?\nWhat’s the relative importance of each hyperparameter?\n\nSome of the users who might see benefit from “opening” the black box of hyperparameter optimization:\n\nresearchers that want to better understand learners in practice\nengineers that want to maximize performance or minimize run time\nteachers that want to demonstrate what happens when tuning hyperparameters\n\nWe’ll use Pima Indians dataset in this blog post, where we want to predict whether or not someone has diabetes, so we’ll perform classification, but the methods we discuss also work for regression and clustering.\nPerhaps we decide we want to try kernlab’s svm for our classification task. Knowing that svm has several hyperparameters to tune, we can ask mlr to list the hyperparameters to refresh our memory:\n\nlibrary(mlr)\nlibrary(ggplot2)\n# to make sure our results are replicable we set the seed\nset.seed(7)\ngetParamSet(\"classif.ksvm\")\n## Type len Def\n## scaled logical - TRUE\n## type discrete - C-svc\n## kernel discrete - rbfdot\n## C numeric - 1\n## nu numeric - 0.2\n## epsilon numeric - 0.1\n## sigma numeric - -\n## degree integer - 3\n## scale numeric - 1\n## offset numeric - 1\n## order integer - 1\n## tol numeric - 0.001\n## shrinking logical - TRUE\n## class.weights numericvector -\n## fit logical - TRUE\n## cache integer - 40\n## Constr Req Tunable Trafo\n## scaled - - TRUE -\n## type C-svc,nu-svc,C-bsvc,spoc-svc,kbb-svc - TRUE -\n## kernel vanilladot,polydot,rbfdot,tanhdot,lap... - TRUE -\n## C 0 to Inf Y TRUE -\n## nu 0 to Inf Y TRUE -\n## epsilon -Inf to Inf Y TRUE -\n## sigma 0 to Inf Y TRUE -\n## degree 1 to Inf Y TRUE -\n## scale 0 to Inf Y TRUE -\n## offset -Inf to Inf Y TRUE -\n## order -Inf to Inf Y TRUE -\n## tol 0 to Inf - TRUE -\n## shrinking - - TRUE -\n## class.weights 0 to Inf - TRUE -\n## fit - - FALSE -\n## cache 1 to Inf - TRUE -\n\nNoting that we have default values for each of the hyperparameters, we could simply accept the defaults for each of the hyperparameters and evaluate our mmce performance using 3-fold cross validation:\n\nrdesc = makeResampleDesc(\"CV\", iters = 3)\nr = resample(\"classif.ksvm\", pid.task, rdesc)\nprint(r)\n## Resample Result\n## Task: PimaIndiansDiabetes-example\n## Learner: classif.ksvm\n## Aggr perf: mmce.test.mean=0.2434896\n## Runtime: 0.337791\n\nWhile this result may seem decent, we have a nagging doubt: what if we chose hyperparameter values different from the defaults? Would we get better results?\nMaybe we believe that the default of kernel = \"rbfdot\" will work well based on our prior knowledge of the dataset, but we want to try altering our regularization to get better performance. For kernlab’s svm, regularization is represented using the C hyperparameter. Calling getParamSet again to refresh our memory, we see that C defaults to 1.\n\ngetParamSet(\"classif.ksvm\")\n## Type len Def\n## scaled logical - TRUE\n## type discrete - C-svc\n## kernel discrete - rbfdot\n## C numeric - 1\n## nu numeric - 0.2\n## epsilon numeric - 0.1\n## sigma numeric - -\n## degree integer - 3\n## scale numeric - 1\n## offset numeric - 1\n## order integer - 1\n## tol numeric - 0.001\n## shrinking logical - TRUE\n## class.weights numericvector -\n## fit logical - TRUE\n## cache integer - 40\n## Constr Req Tunable Trafo\n## scaled - - TRUE -\n## type C-svc,nu-svc,C-bsvc,spoc-svc,kbb-svc - TRUE -\n## kernel vanilladot,polydot,rbfdot,tanhdot,lap... - TRUE -\n## C 0 to Inf Y TRUE -\n## nu 0 to Inf Y TRUE -\n## epsilon -Inf to Inf Y TRUE -\n## sigma 0 to Inf Y TRUE -\n## degree 1 to Inf Y TRUE -\n## scale 0 to Inf Y TRUE -\n## offset -Inf to Inf Y TRUE -\n## order -Inf to Inf Y TRUE -\n## tol 0 to Inf - TRUE -\n## shrinking - - TRUE -\n## class.weights 0 to Inf - TRUE -\n## fit - - FALSE -\n## cache 1 to Inf - TRUE -\n\nLet’s tell mlr to randomly pick C values between 2^-5 and 2^5, evaluating mmce using 3-fold cross validation:\n\n# create the C parameter in continuous space: 2^-5 : 2^5\nps = makeParamSet(\n makeNumericParam(\"C\", lower = -5, upper = 5, trafo = function(x) 2^x)\n)\n# random search in the space with 100 iterations\nctrl = makeTuneControlRandom(maxit = 100L)\n# 3-fold CV\nrdesc = makeResampleDesc(\"CV\", iters = 2L)\n# run the hyperparameter tuning process\nres = tuneParams(\"classif.ksvm\", task = pid.task, control = ctrl,\n resampling = rdesc, par.set = ps, show.info = FALSE)\nprint(res)\n## Tune result:\n## Op. pars: C=0.506\n## mmce.test.mean=0.2213542\n\nmlr gives us the best performing value for C, and we can see that we’ve improved our results vs. just accepting the default value for C. This functionality is available in other machine learning packages, like sci-kit learn’s random search, but this functionality is essentially treating our choice of C as a black box method: we give a search strategy and just accept the optimal value. What if we wanted to get a sense of the relationship between C and mmce? Maybe the relationship is linear in a certain range and we can exploit this to get better even performance! mlr provides 2 methods to help answer this question: generateHyperParsEffectData to generate the resulting data and plotHyperParsEffect providing many options built-in for the user to plot the data.\nLet’s investigate the results from before where we tuned C:\n\ndata = generateHyperParsEffectData(res)\nplotHyperParsEffect(data, x = \"C\", y = \"mmce.test.mean\")\n\n\n\n\nFrom the scatterplot, it appears our optimal performance is somewhere in the region between 2^-2.5 and 2^-1.75. This could provide us a region to further explore if we wanted to try to get even better performance!\nWe could also evaluate how “long” it takes us to find that optimal value:\n\nplotHyperParsEffect(data, x = \"iteration\", y = \"mmce.test.mean\")\n\n\n\n\nBy default, the plot only shows the global optimum, so we can see that we found the “best” performance in less than 25 iterations!\nBut wait, I hear you saying. I also want to tune sigma, the inverse kernel width of the radial basis kernel function. So now we have 2 hyperparameters that we want to simultaneously tune: C and sigma.\n\n# create the C and sigma parameter in continuous space: 2^-5 : 2^5\nps = makeParamSet(\n makeNumericParam(\"C\", lower = -5, upper = 5, trafo = function(x) 2^x),\n makeNumericParam(\"sigma\", lower = -5, upper = 5, trafo = function(x) 2^x)\n)\n# random search in the space with 100 iterations\nctrl = makeTuneControlRandom(maxit = 100L)\n# 3-fold CV\nrdesc = makeResampleDesc(\"CV\", iters = 2L)\n# run the hyperparameter tuning process\nres = tuneParams(\"classif.ksvm\", task = pid.task, control = ctrl,\n resampling = rdesc, par.set = ps, show.info = FALSE)\nprint(res)\n## Tune result:\n## Op. pars: C=0.709; sigma=0.068\n## mmce.test.mean=0.2330729\n# collect the hyperparameter data\ndata = generateHyperParsEffectData(res)\n\nWe can use plotHyperParsEffect to easily create a heatmap with both hyperparameters. We get tons of functionality for free here. For example, mlr will automatically interpolate the grid to get an estimate for values we didn’t even test! All we need to do is pass a regression learner to the interpolate argument:\n\nplotHyperParsEffect(data, x = \"C\", y = \"sigma\", z = \"mmce.test.mean\",\n plot.type = \"heatmap\", interpolate = \"regr.earth\")\n\n\n\n\nIf we use the show.experiments argument, we can see which points were actually tested and which were interpolated:\n\nplotHyperParsEffect(data, x = \"C\", y = \"sigma\", z = \"mmce.test.mean\",\n plot.type = \"heatmap\", interpolate = \"regr.earth\",\n show.experiments = TRUE)\n\n\n\n\nplotHyperParsEffect returns a ggplot2 object, so we can always customize it to better fit our needs downstream:\n\nplt = plotHyperParsEffect(data, x = \"C\", y = \"sigma\", z = \"mmce.test.mean\",\n plot.type = \"heatmap\", interpolate = \"regr.earth\",\n show.experiments = TRUE)\nmin_plt = min(plt$data$mmce.test.mean, na.rm = TRUE)\nmax_plt = max(plt$data$mmce.test.mean, na.rm = TRUE)\nmean_plt = mean(c(min_plt, max_plt))\nplt + scale_fill_gradient2(breaks = seq(min_plt, max_plt, length.out = 4),\n low = \"red\", mid = \"white\", high = \"blue\", midpoint = mean_plt)\n\n\n\n\nNow we can get a good sense of where the separation happens for each of the hyperparameters: in this particular example, we want lower values for sigma and values around 1 for C.\nThis was just a taste of mlr’s hyperparameter tuning visualization capabilities. For the full tutorial, check out the mlr tutorial.\nSome features coming soon:\n\n“Prettier” plot defaults\nSupport for more than 2 hyperparameters\nDirect support for hyperparameter “importance”\n\nThanks to the generous sponsorship from GSoC, and many thanks to my mentors Bernd Bischl and Lars Kotthoff!" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html", - "title": "mlr3 package updates - q1/2022", - "section": "", - "text": "Due to the high amount of packages in the mlr3 ecosystem, it is hard to keep up with the latest changes across all packages. This posts gives an overview by listing the recent release notes of mlr3 packages from the last quarter. Note that only CRAN packages are listed here and the sort order is alphabetically." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section", - "title": "mlr3 package updates - q1/2022", - "section": "0.5.2", - "text": "0.5.2\n\nrefactor: The $print() method of OptimInstance omits unnecessary columns now.\nfix: The $clear() method of OptimInstance raised an error.\nfix: The $clear() method of Archive missed to reset the $start_time field.\nfeat: Optimizer and Terminator objects have the optional field $label now.\nfeat: as.data.table() functions for objects of class Dictionary have been extended with additional columns.\nfeat: Add a as.data.table.DictionaryTerminator() function." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-1", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-1", - "title": "mlr3 package updates - q1/2022", - "section": "0.5.1", - "text": "0.5.1\n\nfix: The return of the $.status() method of TerminatorRunTime and TerminatorClockTime was not in a consistent unit. The return is in seconds from now on.\nfix: The number of evaluations was recorded as 0 in the log messages when the search space was empty.\nfeat: Add a as.data.table.DictionaryOptimizer() function.\nfeat: New $help() method which opens the manual page of an Optimizer." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-2", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-2", - "title": "mlr3 package updates - q1/2022", - "section": "0.5.0", - "text": "0.5.0\n\nfeat: Add $nds_selection() method to Archive.\nfeat: New Codomain class that allows extra parameters.\nrefactor: Objective values were automatically named. From now on, only unnamed returns of ObjectiveRFun are named.\nfix: OptimInstance, Archive and Objective objects were not cloned properly.\nrefactor: The fields $param_classes, $properties and $packages of Optimizer objects are read-only now.\nfeat: The branin() function is exported now." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-3", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-3", - "title": "mlr3 package updates - q1/2022", - "section": "0.13.3", - "text": "0.13.3\n\nMost objects now have a new (optional) field label, i.e. Task, TaskGenerator, Learner, Resampling, and Measure.\nas.data.table() methods for objects of class Dictonary have been extended with additional columns.\nas_task_classif.formula() and as_task_regr.formula() now remove additional atrributes attached to the data which caused some some learners to break.\nPackages are now loaded prior to calling the $train() and $predict() methods of a Learner. This ensures that package loading errors are properly propagated and not affected by encapsulation (#771)." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-4", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-4", - "title": "mlr3 package updates - q1/2022", - "section": "0.13.2", - "text": "0.13.2\n\nSetting a fallback learner for a learner with encapsulation in its default settings now automatically sets encapsulation to \"evaluate\" (#763).\nas_task_classif() and as_task_regr() now support the construction of tasks using the formula interface, e.g. as_task_regr(mpg ~ ., data = mtcars) (#761).\nThe row role \"validation\" has been renamed to \"holdout\". In the next release, mlr3 will start switching to the now more common terms \"train\"/\"validation\" instead of \"train\"/\"test\" for the sets created during resampling." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-5", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-5", - "title": "mlr3 package updates - q1/2022", - "section": "0.13.1", - "text": "0.13.1\n\nImproved performance for many operations on ResampleResult and BenchmarkResult.\nresample() and benchmark() got a new argument clone to control which objects to clone before performing computations.\nTasks are checked for infinite values during the conversion from data.frame to Task in as_task_classif() and as_task_regr(). A warning is signaled if any column contains infinite values." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-6", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-6", - "title": "mlr3 package updates - q1/2022", - "section": "0.5.0", - "text": "0.5.0\n\nAdd references to benchmark paper and praznik paper (#104)\nNew filter FilterSelectedFeatures which makes use of embedded feature selection methods of learners. See the help page for more details (#102)\nAllow NA as task type. This makes it possible to use other tasks than \"regr\" or \"classif\" for certain filters, e.g. FilterVariance (#106)" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-7", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-7", - "title": "mlr3 package updates - q1/2022", - "section": "0.7.0", - "text": "0.7.0\n\nfeat: Allow to pass FSelector objects as method in fselect() and auto_fselector().\nfeat: Added $label to FSelectors.\ndocs: New examples with fselect() function.\nfeat: $help() method which opens manual page of a FSelector.\nfeat: Added a as.data.table.DictionaryFSelector function.\nfeat: Added min_features parameter to FSelectorSequential." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-8", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-8", - "title": "mlr3 package updates - q1/2022", - "section": "0.6.1", - "text": "0.6.1\n\nAdd store_models flag to fselect().\nRemove store_x_domain flag." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-9", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-9", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.0", - "text": "0.4.0\n\nfeat: New adjust_minimum_budget flag in OptimizerSuccessiveHalving. The minimum budget is adjusted in the base stage to use the maximum budget in last stage.\nfeat: New repetitions parameter to specify the exact number of repetitions. Replaced the repeats parameter." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-10", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-10", - "title": "mlr3 package updates - q1/2022", - "section": "0.3.0", - "text": "0.3.0\n\nfeat: TunerHyperband evaluates configurations of same budget across brackets in parallel now.\nfeat: New repeats parameter to repeat runs of successive halving and hyperband until termination.\nfix: Bug where maximization measures were minimized." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-11", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-11", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.9", - "text": "0.4.9\n\nFixed bug in surv.logloss causing IPCW weighting to not be applied correctly" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-12", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-12", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.8", - "text": "0.4.8\n\nBug fixes in AUC measures" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-13", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-13", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.7", - "text": "0.4.7\n\nAdd right-censored log loss\nFix bug in {rpart} where model was being discarded when set to be kept. Parameter model now called keep_model." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-14", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-14", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.6", - "text": "0.4.6\n\nPatch for upstream breakages\nAdd TaskSurv$kaplan method\n{survivalmodels} now imported (previously suggested)" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-15", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-15", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.5", - "text": "0.4.5\n\nImproved reduction from survival matrix predictions to ranking predictions\nFixed cindex bug when all predictions equal\nFix for valgrind" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-16", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-16", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.4", - "text": "0.4.4\n\nMinor change to how distributions are created to better support improper distributions\nFixed bug in simsurv task that made it impossible to predict the target" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-17", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-17", - "title": "mlr3 package updates - q1/2022", - "section": "0.4.3", - "text": "0.4.3\n\nMassive speed-up in distrcompositor PipeOp/pipeline\nMore informative error given if $distr called for a learner that does not support this return type\nFix massive bottleneck in scoring rule measures\nAdd Density coercions as_task_dens and as_prediction_dens\nMeasures now use parameter sets like learners. This streamlines the interface but unfortunately means ids can no longer be set dynamically.\nAdd parameters t_max and p_max to Graf, Schmid and Integrated Log-loss as an alternative to times. t_max is equivalent to times = seq(t_max) and p_max is the proportion of censoring to integrate up to in the dataset.\nFix bug in Rcpp code that was causing erroneous values for calculating the cindex in datasets greater than 20,000 observations." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-18", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-18", - "title": "mlr3 package updates - q1/2022", - "section": "0.1.2", - "text": "0.1.2\n\nrefactor: stars objects are directly converted to terra objects now." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-19", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-19", - "title": "mlr3 package updates - q1/2022", - "section": "0.1.1", - "text": "0.1.1\n\nfix: compatibility to terra update." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-20", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-20", - "title": "mlr3 package updates - q1/2022", - "section": "1.0.1", - "text": "1.0.1\n\nFixed a issue which caused coordinates to appear in the feature set when a data.frame was supplied (#166, @be-marc)\nAdd autoplot() support for \"groups\" column role in rsmp(\"cv\")" - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-21", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-21", - "title": "mlr3 package updates - q1/2022", - "section": "0.13.0", - "text": "0.13.0\n\nfeat: Allow to pass Tuner objects as method in tune() and auto_tuner().\ndocs: Link Tuner to help page of bbotk::Optimizer.\nfeat: Tuner objects have the optional field $label now.\nfeat: as.data.table() functions for objects of class Dictionary have been extended with additional columns." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-22", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-22", - "title": "mlr3 package updates - q1/2022", - "section": "0.12.1", - "text": "0.12.1\n\nfeat: Add a as.data.table.DictionaryTuner function.\nfeat: New $help() method which opens the manual page of an Tuner." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-23", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-23", - "title": "mlr3 package updates - q1/2022", - "section": "0.12.0", - "text": "0.12.0\n\nfeat: as_search_space() function to create search spaces from Learner and ParamSet objects. Allow to pass TuningSpace objects as search_space in TuningInstanceSingleCrit and TuningInstanceMultiCrit.\nfeat: The mlr3::HotstartStack can now be removed after tuning with the keep_hotstart_stack flag.\nfeat: The Archive stores errors and warnings of the learners.\nfeat: When no measure is provided, the default measure is used in auto_tuner() and tune_nested()." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-24", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-24", - "title": "mlr3 package updates - q1/2022", - "section": "0.11.0", - "text": "0.11.0\n\nfix: $assign_result() method in TuningInstanceSingleCrit when search space is empty.\nfeat: Default measure is used when no measure is supplied to TuningInstanceSingleCrit." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-25", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-25", - "title": "mlr3 package updates - q1/2022", - "section": "0.10.0", - "text": "0.10.0\n\nFixes bug in TuningInstanceMultiCrit$assign_result().\nHotstarting of learners with previously fitted models.\nRemove deep clones to speed up tuning.\nAdd store_models flag to auto_tuner().\nAdd \"noisy\" property to ObjectiveTuning." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-26", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-26", - "title": "mlr3 package updates - q1/2022", - "section": "0.2.0", - "text": "0.2.0\n\nfeat: Add a as.data.table.TuningSpace() function.\nfeat: TuningSpace objects have the optional field $label now.\nfeat: New $help() method which opens the manual page of a TuningSpace.\nfeat: Add search space for glmnet and kknn to default collection.\nfeat: New as_search_space() function to create search spaces from TuningSpace objects." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-27", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-27", - "title": "mlr3 package updates - q1/2022", - "section": "0.1.1", - "text": "0.1.1\n\nfix: The subsample hyperparameter is tuned on a logarithmic scale now. The lower bound of alpha is reduced from 1e-4 to 1e-3. The tuning range of the lambda hyperparameter was 0.1 to 1. From now on, lambda is tuned from 1e-3 to 1e3 on a logarithmic scale." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-28", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-28", - "title": "mlr3 package updates - q1/2022", - "section": "0.1.0", - "text": "0.1.0\n\nrefactor: update citations.\nfeat: Add mtry.ratio hyperparameter to tuning spaces of the ranger learner.\nfeat: Add $print() method to TuningSpace objects." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-29", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-29", - "title": "mlr3 package updates - q1/2022", - "section": "0.2.4", - "text": "0.2.4\n\nUpdated reexports." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-30", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-30", - "title": "mlr3 package updates - q1/2022", - "section": "0.5.8", - "text": "0.5.8\n\nCompatibility fixes." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-31", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-31", - "title": "mlr3 package updates - q1/2022", - "section": "0.9.0", - "text": "0.9.0\n\nAdded default_values() function to extract default values from ParamSet objects." - }, - { - "objectID": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-32", - "href": "posts/2022-04-25-mlr3-package-updates-q12022/mlr3-package-updates-q12022.html#section-32", - "title": "mlr3 package updates - q1/2022", - "section": "0.8.0", - "text": "0.8.0\n\nParameters now have a new (optional) field description.\nImproved printing of parameters in documentation (#355).\nA warning is now signaled if the package ParamHelpers is also loaded.\nFixed some links." - }, - { - "objectID": "posts/2016-10-20-paper-published-mlr-machine-learning-in-r/2016-10-20-paper-published-mlr-machine-learning-in-r.html", - "href": "posts/2016-10-20-paper-published-mlr-machine-learning-in-r/2016-10-20-paper-published-mlr-machine-learning-in-r.html", - "title": "Paper published: mlr - Machine Learning in R", - "section": "", - "text": "We are happy to announce that we can finally answer the question on how to cite mlr properly in publications.\nOur paper on mlr has been published in the open-access Journal of Machine Learning Research (JMLR) and can be downloaded on the journal home page.\n\n\n\n\n\nThe paper gives a brief overview of the features of mlr and also includes a comparison with similar toolkits. For an in-depth understanding we still recommend our excellent online mlr tutorial which is now also available as a PDF on arxiv.org or as zipped HTML files for offline reading.\nOnce mlr 2.10 hits CRAN you can retrieve the citation information from within R:\n\ncitation(\"mlr\")\n\n\n## \n## To cite package 'mlr' in publications use:\n## \n## Bischl B, Lang M, Kotthoff L, Schiffner J, Richter J, Studerus E,\n## Casalicchio G, Jones Z (2016). \"mlr: Machine Learning in R.\" _Journal\n## of Machine Learning Research_, *17*(170), 1-5.\n## .\n## \n## Lang M, Kotthaus H, Marwedel P, Weihs C, Rahnenfuehrer J, Bischl B\n## (2014). \"Automatic model selection for high-dimensional survival\n## analysis.\" _Journal of Statistical Computation and Simulation_,\n## *85*(1), 62-76.\n## \n## Bischl B, Kuehn T, Szepannek G (2016). \"On Class Imbalance Correction\n## for Classification Algorithms in Credit Scoring.\" In _Operations\n## Research Proceedings 2014_, 37-43. Springer.\n## \n## Bischl B, Richter J, Bossek J, Horn D, Thomas J, Lang M (2017).\n## \"mlrMBO: A Modular Framework for Model-Based Optimization of\n## Expensive Black-Box Functions.\" _arXiv preprint arXiv:1703.03373_.\n## \n## Probst P, Au Q, Casalicchio G, Stachl C, Bischl B (2017). \"Multilabel\n## Classification with R Package mlr.\" _arXiv preprint\n## arXiv:1703.08991_.\n## \n## Casalicchio G, Bossek J, Lang M, Kirchhoff D, Kerschke P, Hofner B,\n## Seibold H, Vanschoren J, Bischl B (2017). \"OpenML: An R package to\n## connect to the machine learning platform OpenML.\" _Computational\n## Statistics_, 1-15.\n## \n## To see these entries in BibTeX format, use 'print(,\n## bibtex=TRUE)', 'toBibtex(.)', or set\n## 'options(citation.bibtex.max=999)'.\n##" - }, - { - "objectID": "posts/2018-02-28-mlr-training-courses/2018-02-28-mlr-training-courses.html", - "href": "posts/2018-02-28-mlr-training-courses/2018-02-28-mlr-training-courses.html", - "title": "Training Courses for mlr: Machine Learning in R", - "section": "", - "text": "The mlr: Machine Learning in R package provides a generic, object-oriented and extensible framework for classification, regression, survival analysis and clustering for the statistical programming language R. The package targets practitioners who want to quickly apply machine learning algorithms, as well as researchers who want to implement, benchmark, and compare their new methods in a structured environment.\nWe are happy to announce that we now offer training courses specialized on mlr:\n\nThe Munich R Courses already offer the three-day course “Machine Learning and Data Mining in R”. The course includes a basic introduction to theoretical concepts of machine learning and especially focuses on the mlr package. The next course starts on May 2nd, 2018 and will be held in German (see here).\nThe Professional Certificate Program “Data Science” also includes a short introduction to the mlr package. It is a brand new extra-occupational 10-day training at the University of Munich (LMU). The certificate program starts in May 2018 and the application deadline is March 15th, 2018.\nYou can request an inhouse R course in English or German via our contact form.\nIf you offer R or Data Science courses and want to include a course on mlr: Machine Learning in R into your course program, feel free to contact us at rkurse@stat.uni-muenchen.de." - }, - { - "objectID": "posts/2016-08-15-the-mlr-workshop/2016-08-15-the-mlr-workshop.html", - "href": "posts/2016-08-15-the-mlr-workshop/2016-08-15-the-mlr-workshop.html", - "title": "Result of the mlr summer workshop in Palermo", - "section": "", - "text": "The mlr developer team is quite international: Germany, USA, Canada. The time difference between these countries sometimes makes it hard to communicate and develop new features.\nThe idea for this workshop or sprint was to have the possibility to talk about the project status, future and structure, exterminate imperishable bugs and start developing some fancy features.\nSince we wanted to meet at a nice place we decided to go to Palermo, where the department of statistics provided us with a room in the university for the workshop.\nTwelve people from the developer team met from the 8. to 15. August to work on and with mlr." - }, - { - "objectID": "posts/2016-08-15-the-mlr-workshop/2016-08-15-the-mlr-workshop.html#result-of-the-workhop", - "href": "posts/2016-08-15-the-mlr-workshop/2016-08-15-the-mlr-workshop.html#result-of-the-workhop", - "title": "Result of the mlr summer workshop in Palermo", - "section": "Result of the workhop", - "text": "Result of the workhop\nWe closed a lot of issues and developed new features that we will release with version 2.10 of mlr in the next few days.\nThanks to all sample(participants): Giuseppe Casalicchio, Janek Thomas, Xudong Sun, Jakob Bossek, Bernd Bischl, Jakob Richter, Michel Lang, Philipp Probst, Julia Schiffner, Lars Kotthoff, Zachary Jones, Pascal Kerschke!\nWe also head a great time in a great city aside from the workhop, here are some impressions:" - }, - { - "objectID": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html", - "href": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html", - "title": "Visualization of predictions", - "section": "", - "text": "In this post I want to shortly introduce you to the great visualization possibilities of mlr. Within the last months a lot of work has been put into that field. This post is not a tutorial but more a demonstration of how little code you have to write with mlr to get some nice plots showing the prediction behaviors for different learners.\nFirst we define a list containing all the learners we want to visualize. Notice that most of the mlr methods are able to work with just the string (i.e. \"classif.svm\") to know what learner you mean. Nevertheless you can define the learner more precisely with makeLearner() and set some parameters such as the kernel in this example.\nFirst we define the list of learners we want to visualize." - }, - { - "objectID": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#support-vector-machines", - "href": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#support-vector-machines", - "title": "Visualization of predictions", - "section": "Support Vector Machines", - "text": "Support Vector Machines\nNow lets have a look at the different results and lets start with the SVM with a linear kernel.\n\nplotLearnerPrediction(learner = learners[[1]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\n\nWe can see clearly that in fact the decision boundary is indeed linear. Furthermore the misclassified items are highlighted and a 10-fold cross validation to obtain the mean missclassification error is executed.\nFor the polynomial and the radial kernel the decision boundaries already look a bit more sophisticated:\n\nplotLearnerPrediction(learner = learners[[2]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\nplotLearnerPrediction(learner = learners[[3]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\n\nNote that the intensity of the colors also indicates the certainty of the prediction and that this example is probably a rare case where the linear kernel performs best. although this is likely only the case because we didn’t optimize the parameters for the radial kernel." - }, - { - "objectID": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#quadratic-discriminant-analysis", - "href": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#quadratic-discriminant-analysis", - "title": "Visualization of predictions", - "section": "Quadratic Discriminant Analysis", - "text": "Quadratic Discriminant Analysis\n\nplotLearnerPrediction(learner = learners[[4]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\n\nA well known classificator from the basic course of statistics delivers a similar performance as the SVMs." - }, - { - "objectID": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#random-forest", - "href": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#random-forest", - "title": "Visualization of predictions", - "section": "Random Forest", - "text": "Random Forest\n\nplotLearnerPrediction(learner = learners[[5]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\n\nA completely different picture is generated by the random forest. Here you can see that the whole data set is used to generate the model and as a result it looks like it gives a perfect fit but obviously you wouldn’t use the train data to evaluate your model. And the results of the 10-fold cross validation indicate that the random forest is actually not better then the others." - }, - { - "objectID": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#nearest-neighbour", - "href": "posts/2015-07-28-visualisation-of-predictions/2015-07-28-visualisation-of-predictions.html#nearest-neighbour", - "title": "Visualization of predictions", - "section": "Nearest Neighbour", - "text": "Nearest Neighbour\n\nplotLearnerPrediction(learner = learners[[6]], task = iris.task)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( =\n## \"none\")` instead.\n\n\n\n\nIn the default setting knn just look for ‘k=1’ neighbor and as a result the classifier does not return probabilities but only the class labels." - }, - { - "objectID": "posts/2017-07-19-parameter-tuning-with-mlrhyperopt/2017-07-19-parameter-tuning-with-mlrhyperopt.html", - "href": "posts/2017-07-19-parameter-tuning-with-mlrhyperopt/2017-07-19-parameter-tuning-with-mlrhyperopt.html", - "title": "Parameter tuning with mlrHyperopt", - "section": "", - "text": "Hyperparameter tuning with mlr is rich in options as they are multiple tuning methods:\n\nSimple Random Search\nGrid Search\nIterated F-Racing (via irace)\nSequential Model-Based Optimization (via mlrMBO)\n\nAlso the search space is easily definable and customizable for each of the 60+ learners of mlr using the ParamSets from the ParamHelpers Package.\nThe only drawback and shortcoming of mlr in comparison to caret in this regard is that mlr itself does not have defaults for the search spaces. This is where mlrHyperopt comes into play.\nmlrHyperopt offers\n\ndefault search spaces for the most important learners in mlr,\nparameter tuning in one line of code,\nand an API to add and access custom search spaces from the mlrHyperopt Database\n\n\nInstallation\n\n# version >= 1.11 needed.\ndevtools::install_github(\"berndbischl/ParamHelpers\")\ndevtools::install_github(\"jakob-r/mlrHyperopt\", dependencies = TRUE)\n\n\n\nTuning in one line\nTuning can be done in one line relying on the defaults. The default will automatically minimize the missclassification rate.\n\nlibrary(mlrHyperopt)\nres = hyperopt(iris.task, learner = \"classif.svm\")\nres\n## Tune result:\n## Op. pars: cost=3.26e+04; gamma=3.05e-05\n## mmce.test.mean=0.0266667\n\nWe can find out what hyperopt did by inspecting the res object.\nDepending on the parameter space mlrHyperopt will automatically decide for a suitable tuning method:\n\nres$opt.path$par.set\n## Type len Def Constr Req Tunable Trafo\n## cost numeric - 0 -15 to 15 - TRUE Y\n## gamma numeric - -2 -15 to 15 - TRUE Y\nres$control\n## Tune control: TuneControlMBO\n## Same resampling instance: TRUE\n## Imputation value: 1\n## Start: \n## \n## Tune threshold: FALSE\n## Further arguments: list()\n\nAs the search space defined in the ParamSet is only numeric, sequential Bayesian optimization was chosen. We can look into the evaluated parameter configurations and we can visualize the optimization run.\n\ntail(as.data.frame(res$opt.path))\n## cost gamma mmce.test.mean dob eol error.message exec.time\n## 20 4.522371 -5.066062 0.03333333 20 NA 0.031\n## 21 14.998631 -13.678416 0.03333333 21 NA 0.034\n## 22 4.039102 -4.910495 0.04000000 22 NA 0.029\n## 23 12.493414 -10.962122 0.03333333 23 NA 0.029\n## 24 10.410950 -7.232254 0.04000000 24 NA 0.028\n## 25 5.422108 -5.348049 0.03333333 25 NA 0.028\nplotOptPath(res$opt.path)\n## Warning: `guides( = FALSE)` is deprecated. Please use `guides( = \"none\")` instead.\n## `guides( = FALSE)` is deprecated. Please use `guides( = \"none\")` instead.\n## `guides( = FALSE)` is deprecated. Please use `guides( = \"none\")` instead.\n\n\n\n\nThe upper left plot shows the distribution of the tried settings in the search space and contour lines indicate where regions of good configurations are located. The lower right plot shows the value of the objective (the miss-classification rate) and how it decreases over the time. This also shows nicely that wrong settings can lead to bad results.\n\n\nUsing the mlrHyperopt API with mlr\nIf you just want to use mlrHyperopt to access the default parameter search spaces from the Often you don’t want to rely on the default procedures of mlrHyperopt and just incorporate it into your mlr-workflow. Here is one example how you can use the default search spaces for an easy benchmark:\n\nlrns = c(\"classif.xgboost\", \"classif.nnet\")\nlrns = makeLearners(lrns)\ntsk = pid.task\nrr = makeResampleDesc('CV', stratify = TRUE, iters = 10)\nlrns.tuned = lapply(lrns, function(lrn) {\n if (getLearnerName(lrn) == \"xgboost\") {\n # for xgboost we download a custom ParConfig from the Database\n pcs = downloadParConfigs(learner.name = getLearnerName(lrn))\n pc = pcs[[1]]\n } else {\n pc = getDefaultParConfig(learner = lrn)\n }\n ps = getParConfigParSet(pc)\n # some parameters are dependend on the data (eg. the number of columns)\n ps = evaluateParamExpressions(ps,\n dict = mlrHyperopt::getTaskDictionary(task = tsk))\n lrn = setHyperPars(lrn, par.vals = getParConfigParVals(pc))\n ctrl = makeTuneControlRandom(maxit = 20)\n makeTuneWrapper(learner = lrn, resampling = rr, par.set = ps,\n control = ctrl)\n})\nres = benchmark(learners = c(lrns, lrns.tuned), tasks = tsk,\n resamplings = cv10)\nplotBMRBoxplots(res)\n\nAs we can see we were able to improve the performance of xgboost and the nnet without any additional knowledge on what parameters we should tune. Especially for nnet improved performance is noticable.\n\n\nAdditional Information\nSome recommended additional reads\n\nVignette on getting started and also how to contribute by uploading alternative or additional ParConfigs.\nHow to work with ParamSets as part of the Vignette.\nThe slides of the useR 2017 Talk on mlrHyperopt." - }, - { - "objectID": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html", - "href": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html", - "title": "mlr-2.15.0", - "section": "", - "text": "We just released mlr v2.15.0 to CRAN. This version includes some breaking changes and the usual bug fixes from the last three months.\nWe made good progress on the goal of cleaning up the Github repo. We processed nearly all open pull requests (around 40). In the next months we will focus on cleaning up the issue tracker even though most of our time will go into improving the successor package mlr3 and its extension packages.\nUnless there are active contributions from the user side, we do not expect much feature additions for the next version(s) of mlr." - }, - { - "objectID": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html#new-ensemble-filters", - "href": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html#new-ensemble-filters", - "title": "mlr-2.15.0", - "section": "New ensemble filters", - "text": "New ensemble filters\nWith this release it is possible to calculate ensemble filters with mlr [@seijo-pardo2017]. “Ensemble filters” are similar to ensemble models in the way that multiple filters are used to generate the ranking of features. Multiple aggregations functions are supported (min(), mean(), median(), “Borda”) with the latter being the most used one in literature while writing this.\nTo our knowledge there is no other package/framework in R currently that supports ensemble filters in a similar way mlr does. Since mlr makes it possible to use filters from a variety of different packages, the user is able to create powerful ensemble filters. Note however that currently you cannot tune the selection of simple filters since tuning a character vector param is not supported by ParamHelpers. See this discussion for more information.\nHere is a simple toy example how to create ensemble filters in mlr from ?filterFeatures():\n\nlibrary(mlr)\n## Loading required package: ParamHelpers\n## Warning message: 'mlr' is in 'maintenance-only' mode since July 2019.\n## Future development will only happen in 'mlr3'\n## (). Due to the focus on 'mlr3' there might be\n## uncaught bugs meanwhile in {mlr} - please consider switching.\nfilterFeatures(iris.task, method = \"E-min\",\n base.methods = c(\"FSelectorRcpp_gain.ratio\", \"FSelectorRcpp_information.gain\"), abs = 2)\n## Supervised task: iris-example\n## Type: classif\n## Target: Species\n## Observations: 150\n## Features:\n## numerics factors ordered functionals \n## 2 0 0 0 \n## Missings: FALSE\n## Has weights: FALSE\n## Has blocking: FALSE\n## Has coordinates: FALSE\n## Classes: 3Species\n## setosa versicolor virginica \n## 50 50 50 \n## Positive class: NA" - }, - { - "objectID": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html#new-return-structure-for-filter-values", - "href": "posts/2019-08-06-mlr-2-15-0/2019-08-06-mlr-2-15-0.html#new-return-structure-for-filter-values", - "title": "mlr-2.15.0", - "section": "New return structure for filter values", - "text": "New return structure for filter values\nWith the added support for ensemble filters we also changes the return structure of calculated filter values.\nThe new makes it easier to apply post-analysis tasks like grouping and filtering. The “method” of each row is now grouped into one column and the filter values are stored in a separate one. We also added a default sorting of the results by the “value” of each “method”.\nBelow is a comparison of the old and new output:\n\n# new\ngenerateFilterValuesData(iris.task,\n method = c(\"FSelectorRcpp_gain.ratio\", \"FSelectorRcpp_information.gain\"))\n## FilterValues:\n## Task: iris-example\n## name type filter value\n## 1: Petal.Width numeric FSelectorRcpp_gain.ratio 0.8713692\n## 2: Petal.Length numeric FSelectorRcpp_gain.ratio 0.8584937\n## 3: Sepal.Length numeric FSelectorRcpp_gain.ratio 0.4196464\n## 4: Sepal.Width numeric FSelectorRcpp_gain.ratio 0.2472972\n## 5: Petal.Width numeric FSelectorRcpp_information.gain 0.9554360\n## 6: Petal.Length numeric FSelectorRcpp_information.gain 0.9402853\n## 7: Sepal.Length numeric FSelectorRcpp_information.gain 0.4521286\n## 8: Sepal.Width numeric FSelectorRcpp_information.gain 0.2672750\n\n\n# old\ngenerateFilterValuesData(iris.task,\n method = c('gain.ratio','information.gain')\n## FilterValues:\n## Task: iris-example\n## name type gain.ratio information.gain\n## 1 Sepal.Length numeric 0.4196464 0.4521286\n## 2 Sepal.Width numeric 0.2472972 0.2672750\n## 3 Petal.Length numeric 0.8584937 0.9402853\n## 4 Petal.Width numeric 0.8713692 0.9554360" - }, - { - "objectID": "posts/2020-03-30-dalex/2020-03-30-dalex.html", - "href": "posts/2020-03-30-dalex/2020-03-30-dalex.html", - "title": "Explainable machine learning with mlr3 and DALEX", - "section": "", - "text": "Przemysław Biecek and Szymon Maksymiuk added a new chapter to the mlr3 book on how to analyze machine learning models fitted with mlr3 using the excellent DALEX package.\nThe contributed chapter covers an analysis of a random regression forest (implemented in the ranger package) on data extracted from the FIFA video game. In more detail, the following methods for explainable machine learning are showcased:\n\nDataset level exploration: Feature importance and Partial dependency plots.\nInstance level explanation: Break Down, SHapley Additive exPlanations (SHAP), and Ceteris Paribus plots.\n\nHere is a small preview illustrating the effect of different features on the monetary value of Cristiano Ronaldo:\n\nRead the complete chapter here." - }, - { - "objectID": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html", - "href": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html", - "title": "Benchmarking mlr learners on OpenML", - "section": "", - "text": "There are already some benchmarking studies about different classification algorithms out there. The probably most well known and most extensive one is the Do we Need Hundreds of Classifers to Solve Real World Classication Problems? paper. They use different software and also different tuning processes to compare 179 learners on more than 121 datasets, mainly from the UCI site. They exclude different datasets, because their dimension (number of observations or number of features) are too high, they are not in a proper format or because of other reasons. There are also summarized some criticism about the representability of the datasets and the generability of benchmarking results. It remains a bit unclear if their tuning process is done also on the test data or only on the training data (page 3154). They reported the random forest algorithms to be the best one (in general) for multiclass classification datasets and the support vector machine (svm) the second best one. On binary class classification tasks neural networks also perform competitively. They recommend the R library caret for choosing a classifier.\nOther benchmarking studies use much less datasets and are much less extensive (e.g. the Caruana Paper). Computational power was also not the same on these days.\nIn my first approach for benchmarking different learners I follow a more standardized approach, that can be easily redone in future when new learners or datasets are added to the analysis. I use the R package OpenML for getting access to OpenML datasets and the R package mlr (similar to caret, but more extensive) to have a standardized interface to machine learning algorithms in R. Furthermore the experiments are done with the help of the package batchtools, in order to parallelize the experiments (Installation via devtools package: devtools::install_github(“mllg/batchtools”)).\nThe first step is to choose some datasets of the OpenML dataplatform. This is done in the datasets.R file. I want to evaluate classification learners as well as regression learners, so I choose datasets for both tasks. The choosing date was 28.01.2016 so probably nowadays there are more available. I applied several exclusion criteria:\nOf course this exclusion criteria change the representativeness of the datasets.\nThese exclusion criteria provide 184 classification datasets and 98 regression datasets.\nThe benchmark file on these datasets can be found here. For the classification datasets all available classification learners in mlr, that can handle multiclass problems, provide probability estimations and can handle factor features, are used. “boosting” of the adabag package is excluded, because it took too long on our test dataset.\nFor the regression datasets only regression learners that can handle factor features are included. The learners “btgp”, “btgpllm” and “btlm” are excluded, because their training time was too long.\nIn this preliminary study all learners are used with their default hyperparameter settings without tuning. The evaluation technique is 10-fold crossvalidation, 10 times repeated and it is executed by the resample function in mlr. The folds are the same for all the learners. The evaluation measures are the accuracy, the balanced error rate, the (multiclass) auc, the (multiclass) brier score and the logarithmic loss for the classification and the mean square error, mean of absolute error, median of square error and median of absolute error. Additionally the training time is recorded.\nOn 12 cores it took me around 4 days for all datasets.\nI evaluate the results with help of the data.table package, which is good for handling big datasets and fast calculation of subset statistics. Graphics were produced with help of the ggplot package.\nFor comparison, the learners are ranked on each dataset. (see benchmark_analysis.R) There are a few datasets where some of the learners provide errors. In the first approach these were treated as having the worst performance and so all learners providing errors get the worst rank. If there were several learners they get all the averaged worst rank." - }, - { - "objectID": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html#classification", - "href": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html#classification", - "title": "Benchmarking mlr learners on OpenML", - "section": "Classification", - "text": "Classification\nThe results in the classification case, regarding the accuracy are summarized in the following barplot graphic:\n\nIt depicts the average rank regarding accuracy over all classification dataset of each learner.\nClearly the random forest implementations outperform the other. None of the three available packages is clearly better than the other. svm, glmnet and cforest follow. One could probably get better results for svm and xgboost and some other learners with proper tuning.\nThe results for the other measures are quite similar and can be seen here. In the case of the brier score, svm gets the second place and in the logarithmic loss case even the first place. SVM seems to be better suited for these probability measures.\nRegarding training time, kknn, randomForestSRCSyn, naiveBayes and lda gets the best results.\nInstead of taking all datasets one could exclude datasets, where some of the learners got errors. The results are quite similar." - }, - { - "objectID": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html#regression", - "href": "posts/2016-08-11-benchmarking-mlr-learners-on-openml/2016-08-11-benchmarking-mlr-learners-on-openml.html#regression", - "title": "Benchmarking mlr learners on OpenML", - "section": "Regression", - "text": "Regression\nMore interestingly are probably the results of the regression tasks, as there is no available comprehensive regression benchmark study to the best of my knowledge.\nIf an algorithm provided an error it was ranked with the worst rank like in the classification case.\nThe results for the mean squared error can be seen here:\n\nIt depicts the average rank regarding mean square error over all regression dataset of each learner.\nSurprisingly the bartMachine algorithm performs best! The standard random forest implementations are also all under the top 4. cubist, glmnet and kknn also perform very good. The standard linear model (lm) is “unter ferner liefen”.\nbartMachine and cubist are tree based methods combined with an ensembling method like random forest.\nOnce again, if tuning is performed, the ranking would change for algorithms like svm and xgboost.\nResults for the other measures can be seen here. The average rank of cubist gets much better when regarding the mean of absolute error and even gots best, when regarding the median of squared error and median of absolute error. It seems to be a very robust method.\nkknn also gets better for the median of squared and absolute error. Regarding the training time it is once again the unbeaten number one. randomForestSRCSyn is also much faster than the other random forest implementations. lm is also under the best regarding training time.\nWhen omitting datasets where some of the learners produced errors, only 26 regression datasets remain. bartMachine remains best for the mean squared error. The results for the other learners change slightly. See here." - }, - { - "objectID": "posts/2017-03-13-test/index.html", - "href": "posts/2017-03-13-test/index.html", - "title": "First release of mlrMBO - the toolbox for Bayesian Block Box Optimization", - "section": "", - "text": "We are happy to finally announce the first release of mlrMBO on cran after a quite long development time. For the theoretical background and a nearly complete overview of mlrMBOs capabilities you can check our paper on mlrMBO that we presubmitted to arxiv.\nThe key features of mlrMBO are:\n\nGlobal optimization of expensive Black-Box functions.\nMulti-Criteria Optimization.\nParallelization through multi-point proposals.\nSupport for optimization over categorical variables using random forests as a surrogate.\n\nFor examples covering different scenarios we have Vignettes that are also available as an online documentation. For mlr users mlrMBO is especially interesting for hyperparameter optimization.\n\nmlrMBO for mlr hyperparameter tuning was already used in an earlier blog post. Nonetheless we want to provide a small toy example to demonstrate the work flow of mlrMBO in this post.\n\nExample\nFirst, we define an objective function that we are going to minimize:\n\nset.seed(1)\nlibrary(mlrMBO)\n## Loading required package: mlr\n## Loading required package: ParamHelpers\n## Warning message: 'mlr' is in 'maintenance-only' mode since July 2019.\n## Future development will only happen in 'mlr3'\n## (). Due to the focus on 'mlr3' there might be\n## uncaught bugs meanwhile in {mlr} - please consider switching.\n## Loading required package: smoof\n## Loading required package: checkmate\nfun = makeSingleObjectiveFunction(\n name = \"SineMixture\",\n fn = function(x) sin(x[1]) * cos(x[2]) / 2 + 0.04 * sum(x^2),\n par.set = makeNumericParamSet(id = \"x\", len = 2, lower = -5, upper = 5)\n)\n\nTo define the objective function we use makeSingleObjectiveFunction from the neat package smoof, which gives us the benefit amongst others to be able to directly visualize the function. If you happen to be in need of functions to optimize and benchmark your optimization algorithm I recommend you to have a look at the package!\n\nlibrary(plot3D)\nplot3D(fun, contour = TRUE, lightning = TRUE)\n\n\n\n\nLet’s start with the configuration of the optimization:\n\n# In this simple example we construct the control object with the defaults:\nctrl = makeMBOControl()\n# For this numeric optimization we are going to use the Expected\n# Improvement as infill criterion:\nctrl = setMBOControlInfill(ctrl, crit = crit.ei)\n# We will allow for exactly 25 evaluations of the objective function:\nctrl = setMBOControlTermination(ctrl, max.evals = 25L)\n\nThe optimization has to so start with an initial design. mlrMBO can automatically create one but here we are going to use a randomly sampled LHS design of our own:\n\nlibrary(ggplot2)\ndes = generateDesign(n = 8L, par.set = getParamSet(fun),\n fun = lhs::randomLHS)\nautoplot(fun, render.levels = TRUE) + geom_point(data = des)\n\n\n\n\nThe points demonstrate how the initial design already covers the search space but is missing the area of the global minimum. Before we can start the Bayesian optimization we have to set the surrogate learner to Kriging. Therefore we use an mlr regression learner. In fact, with mlrMBO you can use any regression learner integrated in mlr as a surrogate allowing for many special optimization applications.\n\nsur.lrn = makeLearner(\"regr.km\", predict.type = \"se\",\n config = list(show.learner.output = FALSE))\n\nNote: mlrMBO can automatically determine a good surrogate learner based on the search space defined for the objective function. For a purely numeric domain it would have chosen Kriging as well with some slight modifications to make it a bit more stable against numerical problems that can occur during optimization.\nFinally, we can start the optimization run:\n\nres = mbo(fun = fun, design = des, learner = sur.lrn, control = ctrl,\n show.info = TRUE)\n## Computing y column(s) for design. Not provided.\n## [mbo] 0: x=0.897,-2.51 : y = -0.0312 : 0.0 secs : initdesign\n## [mbo] 0: x=-4.52,-0.278 : y = 1.29 : 0.0 secs : initdesign\n## [mbo] 0: x=-2.58,-2.23 : y = 0.63 : 0.0 secs : initdesign\n## [mbo] 0: x=-1.69,1.41 : y = 0.112 : 0.0 secs : initdesign\n## [mbo] 0: x=4.08,4.23 : y = 1.57 : 0.0 secs : initdesign\n## [mbo] 0: x=1.27,-4.52 : y = 0.792 : 0.0 secs : initdesign\n## [mbo] 0: x=-0.163,0.425 : y = -0.0656 : 0.0 secs : initdesign\n## [mbo] 0: x=3.1,3.25 : y = 0.788 : 0.0 secs : initdesign\n## [mbo] 1: x=0.483,-1.18 : y = 0.153 : 0.0 secs : infill_ei\n## [mbo] 2: x=-0.0918,1.51 : y = 0.0885 : 0.0 secs : infill_ei\n## [mbo] 3: x=-0.856,0.593 : y = -0.27 : 0.0 secs : infill_ei\n## [mbo] 4: x=-1.05,-0.239 : y = -0.375 : 0.0 secs : infill_ei\n## [mbo] 5: x=-0.694,-2.25 : y = 0.423 : 0.0 secs : infill_ei\n## [mbo] 6: x=-1.34,0.00144 : y = -0.415 : 0.0 secs : infill_ei\n## [mbo] 7: x=2.3,-2.06 : y = 0.206 : 0.0 secs : infill_ei\n## [mbo] 8: x=-1.55,-0.343 : y = -0.37 : 0.0 secs : infill_ei\n## [mbo] 9: x=1.84,1.01 : y = 0.433 : 0.0 secs : infill_ei\n## [mbo] 10: x=-0.408,4.41 : y = 0.844 : 0.0 secs : infill_ei\n## [mbo] 11: x=5,-2.85 : y = 1.78 : 0.0 secs : infill_ei\n## [mbo] 12: x=-1.29,-0.0751 : y = -0.412 : 0.0 secs : infill_ei\n## [mbo] 13: x=-2.01,-0.0272 : y = -0.291 : 0.0 secs : infill_ei\n## [mbo] 14: x=-5,5 : y = 2.14 : 0.0 secs : infill_ei\n## [mbo] 15: x=-5,-5 : y = 2.14 : 0.0 secs : infill_ei\n## [mbo] 16: x=1.21,3.12 : y = -0.0186 : 0.0 secs : infill_ei\n## [mbo] 17: x=-1.28,0.0491 : y = -0.413 : 0.0 secs : infill_ei\nres$x\n## $x\n## [1] -1.342495094 0.001436926\nres$y\n## [1] -0.4149338\n\nWe can see that we have found the global optimum of \\(y = -0.414964\\) at \\(x = (-1.35265,0)\\) quite sufficiently. Let’s have a look at the points mlrMBO evaluated. Therefore we can use the OptPath which stores all information about all evaluations during the optimization run:\n\nopdf = as.data.frame(res$opt.path)\nautoplot(fun, render.levels = TRUE, render.contours = FALSE) +\n geom_text(data = opdf, aes(label = dob))\n\n\n\n\nIt is interesting to see, that for this run the algorithm first went to the local minimum on the top right in the 6th and 7th iteration but later, thanks to the explorative character of the Expected Improvement, found the real global minimum.\n\n\nComparison\nThat is all good, but how do other optimization strategies perform?\n\nGrid Search\nGrid search is seldom a good idea. But especially for hyperparameter tuning it is still used. Probably because it kind of gives you the feeling that you know what is going on and have not left out any important area of the search space. In reality the grid is usually so sparse that it leaves important areas untouched as you can see in this example:\n\ngrid.des = generateGridDesign(par.set = getParamSet(fun), resolution = 5)\ngrid.des$y = apply(grid.des, 1, fun)\ngrid.des[which.min(grid.des$y), ]\n## x1 x2 y\n## 12 -2.5 0 -0.04923607\nautoplot(fun, render.levels = TRUE, render.contours = FALSE) +\n geom_point(data = grid.des)\n\n\n\n\nIt is no surprise, that the grid search could not cover the search space well enough and we only reach a bad result.\n\n\nWhat about a simple random search?\n\nrandom.des = generateRandomDesign(par.set = getParamSet(fun), n = 25L)\nrandom.des$y = apply(random.des, 1, fun)\nrandom.des[which.min(random.des$y), ]\n## x1 x2 y\n## 20 1.609746 -2.43721 -0.03946573\nautoplot(fun, render.levels = TRUE, render.contours = FALSE) +\n geom_point(data = random.des)\n\n\n\n\nWith the random search you could always be lucky but in average the optimum is not reached if smarter optimization strategies work well.\n\n\nA fair comarison\n… for stochastic optimization algorithms can only be achieved by repeating the runs. mlrMBO is stochastic as the initial design is generated randomly and the fit of the Kriging surrogate is also not deterministic. Furthermore we should include other optimization strategies like a genetic algorithm and direct competitors like rBayesOpt. An extensive benchmark is available in our mlrMBO paper. The examples here are just meant to demonstrate the package.\n\n\n\nEngage\nIf you want to contribute to mlrMBO we ware always open to suggestions and pull requests on github. You are also invited to fork the repository and build and extend your own optimizer based on our toolbox." - }, - { - "objectID": "posts/2020-05-20-mlr3pipelines-webinar/2020-05-20-mlr3pipelines-webinar.html", - "href": "posts/2020-05-20-mlr3pipelines-webinar/2020-05-20-mlr3pipelines-webinar.html", - "title": "Webinar on mlr3pipelines", - "section": "", - "text": "We are happy to be invited to the WhyR Webinar Series to present you with some insights into mlr3pipelines.\nThe authors of mlr3pipelines, Bernd Bischl, Martin Binder and Florian Pfisterer will give you a detailed overview of the vast capabilities of mlr3pipelines, which includes basic data preprocessing operations (like PCA, filtering etc.) but also advanced learner operations, that enable you to build you own stacked learners or ensembles. Combined with tuning methods from mlr3tuning you can build tunable pipelines which effectively gives you the ability to construct AutoML-pipelines.\nThe webinar will be live on the 28th May at 20:00 (CEST / CST+2) on YouTube.\nMake sure to add this event to your calendar or set the reminder on YouTube.\n\n\nWe will update this post once the slides are available." - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html", - "title": "mlr-2.14.0", - "section": "", - "text": "The last mlr release was in August 2018 - so it was definitely time for a new release after around 9 months of development!\nThe NEWS file can be found directly here.\nIn this post we highlight some of the new implementations that come along with this release of v2.14.0" - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#filters", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#filters", - "title": "mlr-2.14.0", - "section": "Filters", - "text": "Filters\nWe integrated the filter methods from the praznik package. These were quite few:\n\npraznik_JMI\npraznik_DISR\npraznik_JMIM\npraznik_MIM\npraznik_NJMIM\npraznik_MRMR\npraznik_CMIM\n\nAlso, a long awaited PR that we finally merged was the inclusion of the FSelectorRcpp filters. These are around 100 times faster than the Java-driven ones from the FSelector package.\nIn addition, we are now using a consistent naming scheme for the filters following _. This change might break your existing code if you used mlr filters before. However, since it is just a naming change we think the burden of updating your code is acceptable." - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#learners", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#learners", - "title": "mlr-2.14.0", - "section": "Learners", - "text": "Learners\nTwo new learners were added:\n\nclassif.liquidSVM\nregr.liquidSVM\n\nLearner regr.h2o.gbm now uses h2o.use.data.table = TRUE by default which should result in a runtime performance increase.\nIt is also possible to retrieve the feature importance of h2O learners now." - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#resampling", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#resampling", - "title": "mlr-2.14.0", - "section": "Resampling", - "text": "Resampling\nYou can now provide fully predefined indices for resampling. This is useful for datasets that have a certain grouping structure (e.g. spatial data) that is difficult to specify otherwise." - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#mlr-org-news", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#mlr-org-news", - "title": "mlr-2.14.0", - "section": "mlr-org NEWS", - "text": "mlr-org NEWS\nYou might be wondering what we’ve been up to in the last months in our group. The major project that we started was mlr3. This is a clean rewrite of mlr with a modular structure to simplify usage and maintenance of the “mlr idea” in the future, both for users and developers. We are not completely finished yet, but you can take a look at the Github repo at what we have achieved so far. Once we are ready to release the initial version, we will of course write a dedicated post about it.\nPutting a lot of time into mlr3 means having less time for responding to issues and questions in mlr. We would like to apologize for this. We are working on this more or less as a side project along our day jobs and our resources are limited. If you want to help and get involved with mlr or mlr3, we would be very happy to have you. Our team is not a closed group and anyone can contribute to the mlr-org projects.\nThe change in development focus also led to a change of maintainer for mlr. As Bernd Bischl (the creator and maintainer) of mlr has a lot of duties, we decided to make Lars Kotthoff and Patrick Schratz the new maintainers of the mlr package.\nmlr will only get bug fixes and minor updates, as we are focusing the development of new things on mlr3. Right now, we have over 400 issues and 30 pull requests so there is a still a lot to do :)" - }, - { - "objectID": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#roadmap-for-mlr", - "href": "posts/2019-04-18-mlr-2-14-0/2019-04-18-mlr-2-14-0.html#roadmap-for-mlr", - "title": "mlr-2.14.0", - "section": "Roadmap for mlr", - "text": "Roadmap for mlr\nWe are will publish new releases every three months from now on, regardless of the changes that have come in. mlr will continue to exist next to mlr3. If users start contributing new features to mlr, we are also happy to include those in the package. As announced already, we will clean up the mlr repo issue and pull request in the coming months to be able to fully concentrate on mlr3 after its initial release." - }, - { - "objectID": "posts/2017-03-02-openml-tutorial-at-user/2017-03-02-openml-tutorial-at-user.html", - "href": "posts/2017-03-02-openml-tutorial-at-user/2017-03-02-openml-tutorial-at-user.html", - "title": "OpenML tutorial at useR!2017 Brussels", - "section": "", - "text": "Conducting research openly and reproducibly is becoming the gold standard in academic research. Practicing open and reproducible research, however, is hard. OpenML.org (Open Machine Learning) is an online platform that aims at making the part of research involving data and analyses easier. It automatically connects data sets, research tasks, algorithms, analyses and results and allows users to access all components including meta information through a REST API in a machine readable and standardized format. Everyone can see, work with and expand other people’s work in a fully reproducible way." - }, - { - "objectID": "posts/2017-03-02-openml-tutorial-at-user/2017-03-02-openml-tutorial-at-user.html#the-user-tutorial", - "href": "posts/2017-03-02-openml-tutorial-at-user/2017-03-02-openml-tutorial-at-user.html#the-user-tutorial", - "title": "OpenML tutorial at useR!2017 Brussels", - "section": "The useR Tutorial", - "text": "The useR Tutorial\nAt useR!2017, we will we will present an R package to interface the OpenML platform and illustrate its usage both as a stand-alone package and in combination with the mlr machine learning package. Furthermore, we show how the OpenML package allows R users to easily search, download and upload machine learning datasets." - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "", - "text": "Tired of learning to use multiple packages to access clustering algorithms?\nUsing different packages makes it difficult to compare the performance of clusterers?\nIt would be great to have just one package that makes interfacing all things clustering easy?\nmlr3cluster to the rescue!\nmlr3cluster is a cluster analysis extention package within the mlr3 ecosystem. It is a successsor of mlr’s cluster capabilities in spirit and functionality.\nIn order to understand the following introduction and tutorial you need to be familiar with R6 and mlr3 basics. See chapters 1-2 of the mlr3book if you need a refresher." - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#installation", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#installation", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Installation", - "text": "Installation\nTo install the package, run the following code chunk:\n\ninstall.packages(\"mlr3cluster\")" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#getting-started", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#getting-started", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Getting Started", - "text": "Getting Started\nAssuming you know all the basics and you’ve installed the package, here’s an example on how to perform k-means clustering on a classic usarrests data set:\n\nlibrary(mlr3)\nlibrary(mlr3cluster)\n\ntask = mlr_tasks$get(\"usarrests\")\nlearner = mlr_learners$get(\"clust.kmeans\")\nlearner$train(task)\npreds = learner$predict(task = task)\n\npreds\n\n for 50 observations:\n row_ids partition\n 1 2\n 2 2\n 3 2\n--- \n 48 1\n 49 1\n 50 1" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-learners", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-learners", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Integrated Learners", - "text": "Integrated Learners\nWhat built-in clusterers does the package come with? Here is a list of integrated learners:\n\nmlr_learners$keys(\"clust\")\n\n [1] \"clust.agnes\" \"clust.ap\" \"clust.cmeans\" \n [4] \"clust.cobweb\" \"clust.dbscan\" \"clust.diana\" \n [7] \"clust.em\" \"clust.fanny\" \"clust.featureless\" \n[10] \"clust.ff\" \"clust.hclust\" \"clust.kkmeans\" \n[13] \"clust.kmeans\" \"clust.MBatchKMeans\" \"clust.meanshift\" \n[16] \"clust.pam\" \"clust.SimpleKMeans\" \"clust.xmeans\" \n\n\nThe library contains all the basic types of clusterers: partitional, hierarchial, density-based and fuzzy. Below is a detailed list of all the learners.\n\n\n\nID\nLearner\nPackage\n\n\n\n\nclust.agnes\nAgglomerative Hierarchical Clustering\ncluster\n\n\nclust.cmeans\nFuzzy C-Means Clustering\ne1071\n\n\nclust.dbscan\nDensity-based Clustering\ndbscan\n\n\nclust.diana\nDivisive Hierarchical Clustering\ncluster\n\n\nclust.fanny\nFuzzy Clustering\ncluster\n\n\nclust.featureless\nSimple Featureless Clustering\nmlr3cluster\n\n\nclust.kmeans\nK-Means Clustering\nstats\n\n\nclust.pam\nClustering Around Medoids\ncluster\n\n\nclust.xmeans\nK-Means with Automatic Determination of k\nRWeka" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-measures", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-measures", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Integrated Measures", - "text": "Integrated Measures\nList of integrated cluster measures:\n\nmlr_measures$keys(\"clust\")\n\n[1] \"clust.ch\" \"clust.db\" \"clust.dunn\" \"clust.silhouette\"\n[5] \"clust.wss\" \n\n\nBelow is a detailed list of all the integrated learners.\n\n\n\nID\nMeasure\nPackage\n\n\n\n\nclust.db\nDavies-Bouldin Cluster Separation\nclusterCrit\n\n\nclust.dunn\nDunn index\nclusterCrit\n\n\nclust.ch\nCalinski Harabasz Pseudo F-Statistic\nclusterCrit\n\n\nclust.silhouette\nRousseeuw’s Silhouette Quality Index\nclusterCrit" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-tasks", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#integrated-tasks", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Integrated Tasks", - "text": "Integrated Tasks\nThere is only one built-in Task in the package:\n\nmlr_tasks$get(\"usarrests\")\n\n (50 x 4): US Arrests\n* Target: -\n* Properties: -\n* Features (4):\n - int (2): Assault, UrbanPop\n - dbl (2): Murder, Rape\n\n\nAs you can see, the biggest difference in clustering tasks as compared to the rest of the tasks in mlr3 is the absense of the Target column." - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#hyperparameters", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#hyperparameters", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Hyperparameters", - "text": "Hyperparameters\nSetting hyperparameters for clusterers is as easy as setting parameters for any other mlr3 learner:\n\ntask = mlr_tasks$get(\"usarrests\")\nlearner = mlr_learners$get(\"clust.kmeans\")\nlearner$param_set\n\n\n id class lower upper nlevels default value\n1: centers ParamUty NA NA Inf 2 2\n2: iter.max ParamInt 1 Inf Inf 10 \n3: algorithm ParamFct NA NA 4 Hartigan-Wong \n4: nstart ParamInt 1 Inf Inf 1 \n5: trace ParamInt 0 Inf Inf 0 \n\nlearner$param_set$values = list(centers = 3L, algorithm = \"Lloyd\", iter.max = 100L)" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#train-and-predict", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#train-and-predict", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Train and Predict", - "text": "Train and Predict\nThe “train” method is simply creating a model with cluster assignments for data, while the “predict” method’s functionality varies depending on the clusterer in question. Read the each learner’s documentation for details.\nFor example, the kmeans learner’s predict method uses clue::cl_predict which performs cluster assignments for new data by looking at the “closest” neighbors of the new observations.\nFollowing the example from the previous section:\n\ntask = mlr_tasks$get(\"usarrests\")\ntrain_set = sample(task$nrow, 0.8 * task$nrow)\ntest_set = setdiff(seq_len(task$nrow), train_set)\n\nlearner = mlr_learners$get(\"clust.kmeans\")\nlearner$train(task, row_ids = train_set)\n\npreds = learner$predict(task, row_ids = test_set)\npreds\n\n for 10 observations:\n row_ids partition\n 9 1\n 13 1\n 18 1\n--- \n 42 1\n 43 1\n 49 2" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#benchmarking-and-evaluation", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#benchmarking-and-evaluation", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Benchmarking and Evaluation", - "text": "Benchmarking and Evaluation\nTo assess the quality of any machine learning experiment, you need to choose an evaluation metric that makes the most sense. Let’s design an experiment that will allow you to compare the performance of three different clusteres on the same task. The mlr3 library provides benchmarking functionality that lets you create such experiments.\n\n# design an experiment by specifying task(s), learner(s), resampling method(s)\ndesign = benchmark_grid(\n tasks = tsk(\"usarrests\"),\n learners = list(\n lrn(\"clust.kmeans\", centers = 3L),\n lrn(\"clust.pam\", k = 3L),\n lrn(\"clust.cmeans\", centers = 3L)),\n resamplings = rsmp(\"holdout\"))\nprint(design)\n\n task learner resampling\n1: \n2: \n3: \n\n# execute benchmark\nbmr = benchmark(design)\n\nINFO [18:07:08.200] [mlr3] Running benchmark with 3 resampling iterations \nINFO [18:07:08.236] [mlr3] Applying learner 'clust.kmeans' on task 'usarrests' (iter 1/1) \nINFO [18:07:08.246] [mlr3] Applying learner 'clust.pam' on task 'usarrests' (iter 1/1) \nINFO [18:07:08.256] [mlr3] Applying learner 'clust.cmeans' on task 'usarrests' (iter 1/1) \nINFO [18:07:08.275] [mlr3] Finished benchmark \n\n# define measure\nmeasures = list(msr(\"clust.silhouette\"))\n\nbmr$aggregate(measures)\n\n nr resample_result task_id learner_id resampling_id iters\n1: 1 usarrests clust.kmeans holdout 1\n2: 2 usarrests clust.pam holdout 1\n3: 3 usarrests clust.cmeans holdout 1\n clust.silhouette\n1: NaN\n2: 0.4536049\n3: 0.5448097" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#visualization", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#visualization", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Visualization", - "text": "Visualization\nHow do you visualize clustering tasks and results? The mlr3viz package (version >= 0.40) now provides that functionality.\n\ninstall.packages(\"mlr3viz\")\n\n\nlibrary(mlr3viz)\n\ntask = mlr_tasks$get(\"usarrests\")\nlearner = mlr_learners$get(\"clust.kmeans\")\nlearner$param_set$values = list(centers = 3L)\nlearner$train(task)\npreds = learner$predict(task)\n\n# Task visualization\nautoplot(task)\n\n\n\n# Pairs plot with cluster assignments\nautoplot(preds, task)\n\n\n\n# Silhouette plot with mean silhouette value as reference line\nautoplot(preds, task, type = \"sil\")\n\n\n\n# Performing PCA on task data and showing cluster assignments\nautoplot(preds, task, type = \"pca\")\n\n\n\n\nKeep in mind that mlr3viz::autoplot also provides more options depending on the kind of plots you’re interested in. For example, to draw borders around clusters, provide appropriate parameters from ggfortify::autoplot.kmeans:\n\nautoplot(preds, task, type = \"pca\", frame = TRUE)\n\n\n\n\nYou can also easily visualize dendrograms:\n\ntask = mlr_tasks$get(\"usarrests\")\nlearner = mlr_learners$get(\"clust.agnes\")\nlearner$train(task)\n\n# Simple dendrogram\nautoplot(learner)\n\n\n\n# More advanced options from `factoextra::fviz_dend`\n# FIXME: https://github.com/mlr-org/mlr3viz/issues/104\n# autoplot(learner,\n# k = learner$param_set$values$k, rect_fill = TRUE,\n# rect = TRUE, rect_border = c(\"red\", \"cyan\"))" - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#further-development", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#further-development", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Further Development", - "text": "Further Development\nIf you have any issues with the package or would like to request a new feature, feel free to open an issue here." - }, - { - "objectID": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#acknowledgements", - "href": "posts/2020-08-26-introducing-mlr3cluster-cluster-analysis-package/introducing-mlr3cluster-cluster-analysis-package.html#acknowledgements", - "title": "Introducing mlr3cluster: Cluster Analysis Package", - "section": "Acknowledgements", - "text": "Acknowledgements\nI would like to thank the following people for their help and guidance: Michel Lang, Lars Kotthoff, Martin Binder, Patrick Schratz, Bernd Bischl." - }, - { - "objectID": "posts/2017-03-09-being-successful-on-kaggle-using-mlr/2017-03-09-being-successful-on-kaggle-using-mlr.html", - "href": "posts/2017-03-09-being-successful-on-kaggle-using-mlr/2017-03-09-being-successful-on-kaggle-using-mlr.html", - "title": "Being successful on Kaggle using mlr", - "section": "", - "text": "Achieving a good score on a Kaggle competition is typically quite difficult. This blog post outlines 7 tips for beginners to improve their ranking on the Kaggle leaderboards. For this purpose, I also created a Kernel for the Kaggle bike sharing competition that shows how the R package, mlr, can be used to tune a xgboost model with random search in parallel (using 16 cores). The R script scores rank 90 (of 3251) on the Kaggle leaderboard." - }, - { - "objectID": "posts/2017-03-09-being-successful-on-kaggle-using-mlr/2017-03-09-being-successful-on-kaggle-using-mlr.html#rules", - "href": "posts/2017-03-09-being-successful-on-kaggle-using-mlr/2017-03-09-being-successful-on-kaggle-using-mlr.html#rules", - "title": "Being successful on Kaggle using mlr", - "section": "7 Rules", - "text": "7 Rules\n\nUse good software\nUnderstand the objective\nCreate and select features\nTune your model\nValidate your model\nEnsemble different models\nTrack your progress\n\n\n1. Use good software\nWhether you choose R, Python or another language to work on Kaggle, you will most likely need to leverage quite a few packages to follow best practices in machine learning. To save time, you should use ‘software’ that offers a standardized and well-tested interface for the important steps in your workflow:\n\nBenchmarking different machine learning algorithms (learners)\nOptimizing hyperparameters of learners\nFeature selection, feature engineering and dealing with missing values\nResampling methods for validation of learner performance\nParallelizing the points above\n\nExamples of ‘software’ that implement the steps above and more:\n\nFor python: scikit-learn (http://scikit-learn.org/stable/auto_examples).\nFor R: mlr (https://mlr.mlr-org.com/index.html) or caret.\n\n\n\n2. Understand the objective\nTo develop a good understanding of the Kaggle challenge, you should:\n\nUnderstand the problem domain:\n\nRead the description and try to understand the aim of the competition.\nKeep reading the forum and looking into scripts/kernels of others, learn from them!\nDomain knowledge might help you (i.e., read publications about the topic, wikipedia is also ok).\nUse external data if allowed (e.g., google trends, historical weather data).\n\nExplore the dataset:\n\nWhich features are numerical, categorical, ordinal or time dependent?\nDecide how to handle missing values. Some options:\n\nImpute missing values with the mean, median or with values that are out of range (for numerical features).\nInterpolate missing values if the feature is time dependent.\nIntroduce a new category for the missing values or use the mode (for categorical features).\n\nDo exploratory data analysis (for the lazy: wait until someone else uploads an EDA kernel).\nInsights you learn here will inform the rest of your workflow (creating new features).\n\n\nMake sure you choose an approach that directly optimizes the measure of interest! Example:\n\nThe median minimizes the mean absolute error (MAE) and the mean minimizes the mean squared error (MSE).\nBy default, many regression algorithms predict the expected mean but there are counterparts that predict the expected median (e.g., linear regression vs. quantile regression). \nFor strange measures: Use algorithms where you can implement your own objective function, see e.g.\n\ntuning parameters of a custom objective or\ncustomize loss function, and evaluation metric.\n\n\n\n\n3. Create and select features:\nIn many kaggle competitions, finding a “magic feature” can dramatically increase your ranking. Sometimes, better data beats better algorithms! You should therefore try to introduce new features containing valuable information (which can’t be found by the model) or remove noisy features (which can decrease model performance):\n\nConcat several columns\nMultiply/Add several numerical columns\nCount NAs per row\nCreate dummy features from factor columns\nFor time series, you could try\n\nto add the weekday as new feature\nto use rolling mean or median of any other numerical feature\nto add features with a lag…\n\nRemove noisy features: Feature selection / filtering\n\n\n\n4. Tune your model\nTypically you can focus on a single model (e.g. xgboost) and tune its hyperparameters for optimal performance.\n\nAim: Find the best hyperparameters that, for the given data set, optimize the pre-defined performance measure.\nProblem: Some models have many hyperparameters that can be tuned.\nPossible solutions:\n\nGrid search or random search\nAdvanced procedures such as irace or mbo (bayesian optimization)\n\n\n\n\n5. Validate your model\nGood machine learning models not only work on the data they were trained on, but also on unseen (test) data that was not used for training the model. When you use training data to make any kind of decision (like feature or model selection, hyperparameter tuning, …), the data becomes less valuable for generalization to unseen data. So if you just use the public leaderboard for testing, you might overfit to the public leaderboard and lose many ranks once the private leaderboard is revealed. A better approach is to use validation to get an estimate of performane on unseen data:\n\nFirst figure out how the Kaggle data was split into train and test data. Your resampling strategy should follow the same method if possible. So if kaggle uses, e.g. a feature for splitting the data, you should not use random samples for creating cross-validation folds.\nSet up a resampling procedure, e.g., cross-validation (CV) to measure your model performance\nImprovements on your local CV score should also lead to improvements on the leaderboard.\nIf this is not the case, you can try\n\nseveral CV folds (e.g., 3-fold, 5-fold, 8-fold)\nrepeated CV (e.g., 3 times 3-fold, 3 times 5-fold)\nstratified CV\n\nmlr offers nice visualizations to benchmark different algorithms.\n\n\n\n6. Ensemble different models (see, e.g. this guide):\nAfter training many different models, you might want to ensemble them into one strong model using one of these methods:\n\nsimple averaging or voting\nfinding optimal weights for averaging or voting\nstacking\n\n\n\n7. Track your progress\nA kaggle project might get quite messy very quickly, because you might try and prototype many different ideas. To avoid getting lost, make sure to keep track of:\n\nWhat preprocessing steps were used to create the data\nWhat model was used for each step\nWhat values were predicted in the test file\nWhat local score did the model achieve\nWhat public score did the model achieve\n\nIf you do not want to use a tool like git, at least make sure you create subfolders for each prototype. This way you can later analyse which models you might want to ensemble or use for your final commits for the competition." - }, - { - "objectID": "posts/2017-03-22-usemlrmbotooptimizeviacommandline/2017-03-22-usemlrmbotooptimizeviacommandline.html", - "href": "posts/2017-03-22-usemlrmbotooptimizeviacommandline/2017-03-22-usemlrmbotooptimizeviacommandline.html", - "title": "Use mlrMBO to optimize via command line", - "section": "", - "text": "Many people who want to apply Bayesian optimization want to use it to optimize an algorithm that is not implemented in R but runs on the command line as a shell script or an executable.\nWe recently published mlrMBO on CRAN. As a normal package it normally operates inside of R, but with this post I want to demonstrate how mlrMBO can be used to optimize an external application. At the same time I will highlight some issues you can likely run into.\nFirst of all we need a bash script that we want to optimize. This tutorial will only run on Unix systems (Linux, OSX etc.) but should also be informative for windows users. The following code will write a tiny bash script that uses bc to calculate \\(sin(x_1-1) + (x_1^2 + x_2^2)\\) and write the result “hidden” in a sentence (The result is 12.34!) in a result.txt text file.\n\nThe bash script\n\n# write bash script\nlines = '#!/bin/bash\nfun ()\n{\n x1=$1\n x2=$2\n command=\"(s($x1-1) + ($x1^2 + $x2^2))\"\n result=$(bc -l <<< $command)\n}\necho \"Start calculation.\"\nfun $1 $2\necho \"The result is $result!\" > \"result.txt\"\necho \"Finish calculation.\"\n'\nwriteLines(lines, \"fun.sh\")\n# make it executable:\nsystem(\"chmod +x fun.sh\")\n\n\n\nRunning the script from R\nNow we need a R function that starts the script, reads the result from the text file and returns it.\n\nlibrary(stringi)\nrunScript = function(x) {\n command = sprintf(\"./fun.sh %f %f\", x[['x1']], x[['x2']])\n error.code = system(command)\n if (error.code != 0) {\n stop(\"Simulation had error.code != 0!\")\n }\n result = readLines(\"result.txt\")\n # the pattern matches 12 as well as 12.34 and .34\n # the ?: makes the decimals a non-capturing group.\n result = stri_match_first_regex(result,\n pattern = \"\\\\d*(?:\\\\.\\\\d+)?(?=\\\\!)\")\n as.numeric(result)\n}\n\nThis function uses stringi and regular expressions to match the result within the sentence. Depending on the output different strategies to read the result make sense. XML files can usually be accessed with XML::xmlParse, XML::getNodeSet, XML::xmlAttrs etc. using XPath queries. Sometimes the good old read.table() is also sufficient. If, for example, the output is written in a file like this:\n\nvalue1 = 23.45\nvalue2 = 13.82\n\nYou can easily use source() like that:\n\nEV = new.env()\neval(expr = {a = 1}, envir = EV)\nas.list(EV)\nsource(file = \"result.txt\", local = EV)\nres = as.list(EV)\nrm(EV)\n\nwhich will return a list with the entries $value1 and $value2.\n\n\nDefine bounds, wrap function.\nTo evaluate the function from within mlrMBO it has to be wrapped in smoof function. The smoof function also contains information about the bounds and scales of the domain of the objective function defined in a ParameterSet.\n\nlibrary(mlrMBO)\n## Loading required package: mlr\n## Loading required package: ParamHelpers\n## Warning message: 'mlr' is in 'maintenance-only' mode since July 2019.\n## Future development will only happen in 'mlr3'\n## (). Due to the focus on 'mlr3' there might be\n## uncaught bugs meanwhile in {mlr} - please consider switching.\n## Loading required package: smoof\n## Loading required package: checkmate\n# Defining the bounds of the parameters:\npar.set = makeParamSet(\n makeNumericParam(\"x1\", lower = -3, upper = 3),\n makeNumericParam(\"x2\", lower = -2.5, upper = 2.5)\n)\n# Wrapping everything in a smoof function:\nfn = makeSingleObjectiveFunction(\n id = \"fun.sh\",\n fn = runScript,\n par.set = par.set,\n has.simple.signature = FALSE\n)\n# let's see if the function is working\ndes = generateGridDesign(par.set, resolution = 3)\ndes$y = apply(des, 1, fn)\ndes\n## x1 x2 y\n## 1 -3 -2.5 16.006802\n## 2 0 -2.5 5.408529\n## 3 3 -2.5 16.159297\n## 4 -3 0.0 9.756802\n## 5 0 0.0 0.841471\n## 6 3 0.0 9.909297\n## 7 -3 2.5 16.006802\n## 8 0 2.5 5.408529\n## 9 3 2.5 16.159297\n\nIf you run this locally, you will see that the console output generated by our shell script directly appears in the R-console. This can be helpful but also annoying.\n\n\nRedirecting output\nIf a lot of output is generated during a single call of system() it might even crash R. To avoid that I suggest to redirect the output into a file. This way no output is lost and the R console does not get flooded. We can simply achieve that by replacing the command in the function runScript from above with the following code:\n\n # console output file output_1490030005_1.1_2.4.txt\n output_file = sprintf(\"output_%i_%.1f_%.1f.txt\",\n as.integer(Sys.time()), x[['x1']], x[['x2']])\n # redirect output with ./fun.sh 1.1 2.4 > output.txt\n # alternative: ./fun.sh 1.1 2.4 > /dev/null to drop it\n command = sprintf(\"./fun.sh %f %f > %s\", x[['x1']], x[['x2']], output_file)\n\n\n\nStart the Optimization\nNow everything is set so we can proceed with the usual MBO setup:\n\nctrl = makeMBOControl()\nctrl = setMBOControlInfill(ctrl, crit = crit.ei)\nctrl = setMBOControlTermination(ctrl, iters = 10)\nconfigureMlr(show.info = FALSE, show.learner.output = FALSE)\nrun = mbo(fun = fn, control = ctrl)\n## Computing y column(s) for design. Not provided.\n## [mbo] 0: x1=-0.976; x2=-1.1 : y = 1.26 : 0.0 secs : initdesign\n## [mbo] 0: x1=0.463; x2=0.116 : y = 0.285 : 0.0 secs : initdesign\n## [mbo] 0: x1=1.23; x2=2.33 : y = 7.19 : 0.0 secs : initdesign\n## [mbo] 0: x1=-0.708; x2=-0.179 : y = 0.458 : 0.0 secs : initdesign\n## [mbo] 0: x1=2.35; x2=0.691 : y = 6.99 : 0.0 secs : initdesign\n## [mbo] 0: x1=-2.57; x2=-1.55 : y = 9.38 : 0.0 secs : initdesign\n## [mbo] 0: x1=-2.05; x2=1.51 : y = 6.41 : 0.0 secs : initdesign\n## [mbo] 0: x1=1.87; x2=-2.31 : y = 9.6 : 0.0 secs : initdesign\n## [mbo] 1: x1=0.00131; x2=-0.273 : y = 0.766 : 0.0 secs : infill_ei\n## [mbo] 2: x1=0.293; x2=2.5 : y = 5.68 : 0.0 secs : infill_ei\n## [mbo] 3: x1=-0.335; x2=0.357 : y = 0.733 : 0.0 secs : infill_ei\n## [mbo] 4: x1=0.287; x2=0.221 : y = 0.523 : 0.0 secs : infill_ei\n## [mbo] 5: x1=-0.515; x2=-0.702 : y = 0.24 : 0.0 secs : infill_ei\n## [mbo] 6: x1=0.446; x2=-0.671 : y = 0.124 : 0.0 secs : infill_ei\n## [mbo] 7: x1=-0.528; x2=-2.5 : y = 5.53 : 0.0 secs : infill_ei\n## [mbo] 8: x1=0.589; x2=-0.535 : y = 0.234 : 0.0 secs : infill_ei\n## [mbo] 9: x1=0.442; x2=-0.525 : y = 0.0586 : 0.0 secs : infill_ei\n## [mbo] 10: x1=0.438; x2=-0.412 : y = 0.172 : 0.0 secs : infill_ei\n# The resulting optimal configuration:\nrun$x\n## $x1\n## [1] 0.4419015\n## \n## $x2\n## [1] -0.5250245\n# The best reached value:\nrun$y\n## [1] 0.05864618\n\n\n\nExecute the R script from a shell\nAlso you might not want to bothered having to start R and run this script manually so what I would recommend is saving all above as an R-script plus some lines that write the output in a JSON file like this:\n\nlibrary(jsonlite)\nwrite_json(run[c(\"x\",\"y\")], \"mbo_res.json\")\n\nLet’s assume we saved all of that above as an R-script under the name runMBO.R (actually it is available as a gist).\nThen you can simply run it from the command line:\n\nRscript runMBO.R\n\nAs an extra the script in the gist also contains a simple handler for command line arguments. In this case you can define the number of optimization iterations and the maximal allowed time in seconds for the optimization. You can also define the seed to make runs reproducible:\n\nRscript runMBO.R iters=20 time=10 seed=3\n\nIf you want to build a more advanced command line interface you might want to have a look at docopt.\n\n\nClean up\nTo clean up all the files generated by this script you can run:\n\nfile.remove(\"result.txt\")\nfile.remove(\"fun.sh\")\nfile.remove(\"mbo_res.json\")\noutput.files = list.files(pattern = \"output_\\\\d+_[0-9_.-]+\\\\.txt\")\nfile.remove(output.files)" - }, - { - "objectID": "posts/2022-02-03-google-summer-of-code-and-mlr3/google-summer-of-code-and-mlr3.html", - "href": "posts/2022-02-03-google-summer-of-code-and-mlr3/google-summer-of-code-and-mlr3.html", - "title": "Google Summer of Code and mlr3", - "section": "", - "text": "Our Org is currently thinking about participating in GSOC 2022 again. We are considering which projects to propose and would also like to collect input. This year’s Google Summer of Code is not only directed at students but at a broader audience of people new to FOSS.\nWe have thought about the following projects:\n\nmlr3multioutput We would like for mlr3multioutput to include multi-label and general multi-output techniques, such as learners and prediction post-processing methods. Furthermore, some work on the evaluation of multi-output methods could drastically improve the package’s utility.\nmlr3fairness by now contains the basic infrastructure to audit algorithms for some notions of fairness. We’d like to augment mlr3fairness with several bias mitigation techniques, as well as other notions of fairness.\nQueue-based asynchronous parallelization for R mlr3 currently heavily relies on future for parallelization. Some tuning methods would profit from a different take on parallelization, where task queues are cleanly orchestrated on a central server. The goal of this project would be to develop a general-purpose parallelization package (maybe extending the future framework) for asynchronous parallelization and parallel programming.\n\nIf you feel like one of the projects sparks your interest, feel free to get in contact with us (mlr-org@stat.uni-muenchen.de)! Similarly, if you would like to propose your own idea/project also get in contact with us, and we will figure out if this is a fit for our organization!" - }, - { - "objectID": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html", - "href": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html", - "title": "Most Popular Learners in mlr", - "section": "", - "text": "For the development of mlr as well as for an “machine learning expert” it can be handy to know what are the most popular learners used. Not necessarily to see, what are the top notch performing methods but to see what is used “out there” in the real world. Thanks to the nice little package cranlogs from metacran you can at least get a slight estimate as I will show in the following…\nFirst we need to install the cranlogs package using devtools:\nNow let’s load all the packages we will need:\nDo obtain a neat table of all available learners in mlr we can call listLearners(). This table also contains a column with the needed packages for each learner separated with a ,.\nNote: You might get some warnings here because you likely did not install all packages that mlr suggests – which is totally fine.\nNow we can obtain the download counts from the rstudio cran mirror, i.e. from the last month. We use data.table to easily sum up the download counts of each day.\nAs some learners need multiple packages we will use the download count of the package with the least downloads.\nLet’s put these numbers in our table:\nHere are the first 5 rows of the table:\nNow let’s get rid of the duplicates introduced by the distinction of the type classif, regr and we already have our…" - }, - { - "objectID": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#nearly-final-table", - "href": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#nearly-final-table", - "title": "Most Popular Learners in mlr", - "section": "Nearly final table", - "text": "Nearly final table\n\nlrns.small = lrns[, .SD[1,], by = .(name, package)]\nlrns.small[, .(class, name, package, downloads)]\n\nThe top 20 according to the rstudio cran mirror:\n\n\n\n\n\n\n\n\n\n\nclass\nname\npackage\ndownloads\n\n\n\n\nclassif.naiveBayes\nNaive Bayes\ne1071\n327427\n\n\nclassif.svm\nSupport Vector Machines (libsvm)\ne1071\n327427\n\n\nclassif.lda\nLinear Discriminant Analysis\nMASS\n202214\n\n\nclassif.qda\nQuadratic Discriminant Analysis\nMASS\n202214\n\n\nsurv.coxph\nCox Proportional Hazard Model\nsurvival\n141819\n\n\nclassif.xgboost\neXtreme Gradient Boosting\nxgboost\n115019\n\n\nclassif.gausspr\nGaussian Processes\nkernlab\n96829\n\n\nclassif.ksvm\nSupport Vector Machines\nkernlab\n96829\n\n\nclassif.lssvm\nLeast Squares Support Vector Machine\nkernlab\n96829\n\n\ncluster.kkmeans\nKernel K-Means\nkernlab\n96829\n\n\nregr.rvm\nRelevance Vector Machine\nkernlab\n96829\n\n\nclassif.ranger\nRandom Forests\nranger\n89234\n\n\nclassif.cvglmnet\nGLM with Lasso or Elasticnet Regularization (Cross Validated Lambda)\nglmnet\n73341\n\n\nclassif.glmnet\nGLM with Lasso or Elasticnet Regularization\nglmnet\n73341\n\n\nsurv.cvglmnet\nGLM with Regularization (Cross Validated Lambda)\nglmnet\n73341\n\n\nsurv.glmnet\nGLM with Regularization\nglmnet\n73341\n\n\nclassif.fnn\nFast k-Nearest Neighbour\nFNN\n72878\n\n\nregr.fnn\nFast k-Nearest Neighbor\nFNN\n72878\n\n\nclassif.randomForest\nRandom Forest\nrandomForest\n69407\n\n\nclassif.rpart\nDecision Tree\nrpart\n49749\n\n\n\n\nAs we are just looking for the packages let’s compress the table a bit further and come to our…" - }, - { - "objectID": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#final-table", - "href": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#final-table", - "title": "Most Popular Learners in mlr", - "section": "Final table", - "text": "Final table\n\nlrns.pgks = lrns[,list(learners = paste(class, collapse = \",\")),\n by = .(package, downloads)]\nlrns.pgks\n\nHere are the first 20 rows of the table:\n\n\n\n\n\n\n\n\n\npackage\ndownloads\nlearners\n\n\n\n\ne1071\n327427\nclassif.naiveBayes,classif.svm,regr.svm\n\n\nMASS\n202214\nclassif.lda,classif.qda\n\n\nsurvival\n141819\nsurv.coxph\n\n\nxgboost\n115019\nclassif.xgboost,regr.xgboost\n\n\nkernlab\n96829\nclassif.gausspr,classif.ksvm,classif.lssvm,cluster.kkmeans,regr.gausspr,regr.ksvm,regr.rvm\n\n\nranger\n89234\nclassif.ranger,regr.ranger,surv.ranger\n\n\nglmnet\n73341\nclassif.cvglmnet,classif.glmnet,regr.cvglmnet,regr.glmnet,surv.cvglmnet,surv.glmnet\n\n\nFNN\n72878\nclassif.fnn,regr.fnn\n\n\nrandomForest\n69407\nclassif.randomForest,regr.randomForest\n\n\nrpart\n49749\nclassif.rpart,regr.rpart,surv.rpart\n\n\ne1071,clue\n39615\ncluster.cmeans\n\n\nklaR\n38043\nclassif.rda\n\n\ngbm\n37986\nclassif.gbm,regr.gbm,surv.gbm\n\n\nnnet\n36318\nclassif.multinom,classif.nnet,regr.nnet\n\n\nclass\n35869\nclassif.knn,classif.lvq1\n\n\nGPfit\n34066\nregr.GPfit\n\n\nh2o\n31550\nclassif.h2o.deeplearning,classif.h2o.gbm,classif.h2o.glm,classif.h2o.randomForest,regr.h2o.deeplearning,regr.h2o.gbm,regr.h2o.glm,regr.h2o.randomForest\n\n\ncaret,pls\n28298\nclassif.plsdaCaret\n\n\npls\n28298\nregr.pcr,regr.plsr\n\n\nRWeka\n23681\nclassif.adaboostm1,classif.IBk,classif.J48,classif.JRip,classif.OneR,classif.PART,cluster.Cobweb,cluster.EM,cluster.FarthestFirst,cluster.SimpleKMeans,cluster.XMeans,regr.IBk\n\n\n\n\nAnd of course we want to have a small visualization:\n\nlibrary(ggplot2)\nlibrary(forcats)\nlrns.pgks$learners = factor(lrns.pgks$learners, lrns.pgks$learners)\ng = ggplot(lrns.pgks[20:1], aes(x = fct_inorder(stri_sub(\n paste0(package,\": \",learners), 0, 64)), y = downloads, fill = downloads))\ng + geom_bar(stat = \"identity\") +\n coord_flip() +\n xlab(\"\") +\n scale_fill_continuous(guide=FALSE)\n## Warning: It is deprecated to specify `guide = FALSE` to remove a guide. Please\n## use `guide = \"none\"` instead." - }, - { - "objectID": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#remarks", - "href": "posts/2017-03-30-mostpopularlearnersinmlr/2017-03-30-mostpopularlearnersinmlr.html#remarks", - "title": "Most Popular Learners in mlr", - "section": "Remarks", - "text": "Remarks\nThis is not really representative of how popular each learner is, as some packages have multiple purposes (e.g. multiple learners). Furthermore it would be great to have access to the trending list. Also most stars at GitHub gives a better view of what the developers are interested in. Looking for machine learning packages we see there e.g: xgboost, h2o and tensorflow." - }, - { - "objectID": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#first-of-all", - "href": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#first-of-all", - "title": "Introducing mlrPlayground", - "section": "First of all", - "text": "First of all\nYou may ask yourself how is this name ‘mlrPlayground’ even justified? What a person dares to put two such opposite terms in a single word and expects people to take him seriously?\nI assume most of you know ‘mlr’, for those who don’t: It is a framework offering a huge variety of tools for simplifying machine learning tasks in R. Quite the opposite from a place, where you can play with your best friends, make new friends, live out your fantasies and just have a great time the whole day until your parents pick you up. Well, for most of the readers here this may not be the case anymore – we know, we are still young in our heart, but let’s be honest … For sure, we all have those memories and definitely have certain associations with the word ‘Playground’. So, what is about this thing called ‘mlrPlayground’?" - }, - { - "objectID": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#the-idea", - "href": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#the-idea", - "title": "Introducing mlrPlayground", - "section": "The idea", - "text": "The idea\nThe idea behind this project was to offer a platform in the form of a Shiny web application, in which a user can try out different kinds of learners provided by the mlr package. On a small set of distinct and tunable regression and classification tasks, it is possible to observe the prediction/performance behavior based on changes on the task, the learner or the learner’s hyperparameters. The user is able to gain new insights and a deeper understanding of how a learner performs, where it’s advantages are and in which cases the learner might fail.\nThere are a lot of different settings we want to offer in the user interface, and so – to not remove the fun of our playground – a huge effort went into creating an aesthetically pleasing and user-friendly UI. To achieve this, a website template was downloaded from Templated and used as the baseline design. After extending the template with missing CSS classes, most of the used shiny widgets have been overwritten – or even completely replaced –, offering refreshingly new visuals for the well-known shiny framework. For the smooth feel of the app, an object-oriented R6 class system with reactive attributes was engineered for the backend to give a well-defined framework of which elements should trigger what evaluation; an otherwise extremely tiresome and error-prone task for dozens of different UI elements.\nAfter all ‘mlrPlayground’ may not be as fun as a real playground, but you are also not as likely to hurt yourself and it is definitely more entertaining than looking at boring pictures of learners in a book." - }, - { - "objectID": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#the-features", - "href": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#the-features", - "title": "Introducing mlrPlayground", - "section": "The features", - "text": "The features\nIn the following, an extended overview of most available features is presented with the help of animated pictures.\n\nTask selection\nWhen running the app and scrolling down you can basically see two big boxes with content: The left one is for the task and learner settings, the right one for an overview of the learner’s behavior. For first-time users, it is advisable (but not necessary) to explore the different tasks. After finding the right button (this task is left to the motivated reader) a panel with all available datasets is extended over our screen. Here, we can choose the task type to receive a varied selection of classification or regression tasks and several other parameters influencing each dataset such as size, noise, and the train/test split.\n \n\n\nLearner selection\nBack at the main panel, we can then select a learner for our task – due to limited manpower not all available learners in mlr made it into this selection. You can expect additions in the future. By making a selection, the main process of the app, consisting of several substeps, gets executed:\n\n\n\na model is trained on the training set of the defined task\n\n\n\n\nperformance measures are calculated on the corresponding test set\n\n\n\n\nthe prediction surface is evaluated and depicted in an interactive plot.\n\n\n\n \n\n\nPerformance measures\nThe first visual change after calculations have finished is the appearance of the prediction plot. Besides, there are now a whole bunch of different performance measures to explore by clicking on the equally named bar on the right-hand side. These provide the opportunity to manually tune each learner on the tasks or compare how each measure behaves in certain scenarios. You will not only be able to understand the learner better, but also each measure. What’s particularly interesting for the latter is the configurable ratio of train and test data in the task selection window.\n \n\n\nHyperparameters\nIf you want to see the effect of hyperparameters on the plot/performance measures, simply click on the small cog next to the learner selection. Now, the left panel is replaced by a window with all available hyperparameters – changes here will have an immediate effect: The whole model will be rerun and all plots/numbers will be updated.\n\n\nSecond learner\nOnly observing changes of a single learner may become a bit boring after time, so an option is available to easily double the fun: Pressing “Add learner” gives the choice of adding a second learner with a second plot, making it possible to compare the prediction surfaces of two different learners on the same task right next of each other. Exciting!\n \nBut wait, there’s still more to offer!\n\n\n\n\nLearning curve and ROC\nThe user can change the created plots on the right panel by switching to the tabs “Learning Curve” (classification and regression) or “ROC” (only classification). The former tells you how well the learner performed or how multiple learners compare to each other. All of this happens with respect to the user selected performance measures (y-axis) on a fraction of the original train data (x-axis). Every additionally selected measure results in an additional plot.\nLast but not least, the ROC curve is only a ROC curve with its default settings for the x- and y-axis: Plotting the true positive rate (TPR) against the false positive rate (FPR). It shows how well the learner can separate the classes (the bigger the area under the curve, the better)." - }, - { - "objectID": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#usage", - "href": "posts/2019-06-23-introducing-mlrplayground/2019-06-23-introducing-mlrplayground.html#usage", - "title": "Introducing mlrPlayground", - "section": "Usage", - "text": "Usage\nIf you want to explore mlrPlayground on your own, you have two possibilities:\nEither visit the website (big thanks to the Department of computational statistics at the LMU Munich for hosting)\nor\ninstall it locally on your own machine using remotes::install_github(\"SebGruber1996/mlrPlayground\") and start with mlrPlayground::start(). The first approach is quicker but may crash if too many people access it at the same time, while the latter requires a local installation. It’s your choice. \nThanks for spending your time reading this blog post instead of being on a real playground! \nSide note: The app is still far from being mature. The UI is – especially for the hyperparameters – not working well on small screens. Also, some hyperparameter settings may cause the app to crash. Even if these things would be solved, there are still a lot of learners missing in the app which are available in the mlr package. Pull requests are welcome at https://github.com/SebGruber1996/mlrPlayground." - }, - { - "objectID": "book.html", - "href": "book.html", - "title": "Manual", - "section": "", - "text": "The “mlr3book” is a free manual written in an online-book style available in two versions.\nIt is the main reference for the mlr3 ecosystem.\n\nHTML\nPDF" - }, - { - "objectID": "contributing.html", - "href": "contributing.html", - "title": "Contributing", - "section": "", - "text": "You are welcome to open a Pull Request on GitHub. Before doing so, please read our PR Guidelines.\nIf you\n\nwant to add a new Learner, PipeOp or Measure you can check out the chapter extending of the mlr3 book.\nare interested in creating a new R package for the mlr3 ecosystem or collaborate on an existing one, you can contact us on Mattermost.\n\nAdditional developer information lives in the mlr3 GitHub Wiki." - }, - { - "objectID": "terminators.html", - "href": "terminators.html", - "title": "", - "section": "", - "text": "Stop tuning when a performance level is reached.\n\nlibrary(mlr3verse)\n\nLoading required package: mlr3\n\n# load terminator and set performance level\nterminator = trm(\"perf_reached\", level = 0.25)\n\n# load tuner\ntuner = tnr(\"random_search\", batch_size = 10)\n\n# retrieve task\ntask = tsk(\"pima\")\n\n# load learner and set search space\nlearner = lts(lrn(\"classif.rpart\"))\n\n# set instance\ninstance = TuningInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = rsmp(\"holdout\"),\n measure = msr(\"classif.ce\"),\n terminator = terminator\n)\n\n# hyperparameter tuning on the pima data set\ntuner$optimize(instance)\n\n minsplit minbucket cp learner_param_vals x_domain classif.ce\n1: 4.021071 3.686677 -4.375425 0.2421875\n\n# best performing hyperparameter configuration\ninstance$result\n\n minsplit minbucket cp learner_param_vals x_domain classif.ce\n1: 4.021071 3.686677 -4.375425 0.2421875\n\n# fit final model on complete data set\nlearner$param_set$values = instance$result_learner_param_vals\nlearner$train(task)\n\nprint(learner)\n\n: Classification Tree\n* Model: rpart\n* Parameters: xval=0, minsplit=55, minbucket=39, cp=0.01258\n* Packages: mlr3, rpart\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, multiclass, selected_features,\n twoclass, weights" - }, - { - "objectID": "filters.html", - "href": "filters.html", - "title": "", - "section": "", - "text": "Feature Filters quantify the importance of each feature of a Task by assigning them a numerical score. In a second step, features can be selected by either selecting a fixed absolute or relative frequency of the best features, or by thresholding on the score value.\nThe Filter PipeOp allows to use filters as a preprocessing step.\n\n\n\n\n\n\n\n\n\nUse the \\(-\\log_10()\\)-transformed \\(p\\)-values of a Kruskal-Wallis rank sum test (implemented in kruskal.test()) for filtering features of the Pima Indian Diabetes tasks.\n\nlibrary(\"mlr3verse\")\n\nLoading required package: mlr3\n\n# retrieve a task\ntask = tsk(\"pima\")\n\n# retrieve a filter\nfilter = flt(\"kruskal_test\")\n\n# calculate scores\nfilter$calculate(task)\n\n# access scores\nfilter$scores\n\n glucose age mass insulin triceps pregnant pedigree pressure \n39.885381 16.942901 16.740864 13.127828 9.158113 7.426955 5.922431 5.788607 \n\n# plot scores\nautoplot(filter)\n\n\n\n# subset task to 3 most important features\ntask$select(head(names(filter$scores), 3))\ntask$feature_names\n\n[1] \"age\" \"glucose\" \"mass\"" - }, - { - "objectID": "support.html", - "href": "support.html", - "title": "Support", - "section": "", - "text": "Questions regarding the software can be asked on Stack Overflow, where you can use the mlr3 tag." - }, - { - "objectID": "support.html#github", - "href": "support.html#github", - "title": "Support", - "section": "GitHub", - "text": "GitHub\nFor bug reports, suggestions, or feature requests please raise an issue in the corresponding GitHub repository. Your issue will be addressed sooner if you provide a reproducible example - e.g. using the reprex R package - and if you open it in the correct repository." - }, - { - "objectID": "support.html#mattermost", - "href": "support.html#mattermost", - "title": "Support", - "section": "Mattermost", - "text": "Mattermost\nYou can join our public Mattermost channel." - }, - { - "objectID": "gallery/2019-11-15-jornadas-de-usarios-de-r/2019-11-15-jornadas-de-usarios-de-r.html", - "href": "gallery/2019-11-15-jornadas-de-usarios-de-r/2019-11-15-jornadas-de-usarios-de-r.html", - "title": "XI Jornadas de Usuarios de R - mlr3", - "section": "", - "text": "Please note, that there have been slight changes in the syntax since the creation of this video." - }, - { - "objectID": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html", - "href": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html", - "title": "Encode Factor Levels for xgboost", - "section": "", - "text": "The package xgboost unfortunately does not support handling of categorical features. Therefore, it is required to manually convert factor columns to numerical dummy features. We show how to use mlr3pipelines to augment the xgboost learner with an automatic factor encoding.\nWe load the mlr3verse package which pulls in the most important packages for this example.\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented." - }, - { - "objectID": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html#construct-the-base-objects", - "href": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html#construct-the-base-objects", - "title": "Encode Factor Levels for xgboost", - "section": "Construct the Base Objects", - "text": "Construct the Base Objects\nFirst, we take an example task with factors (german_credit) and create the xgboost learner:\n\nlibrary(mlr3learners)\n\ntask = tsk(\"german_credit\")\nprint(task)\n\n (1000 x 21): German Credit\n* Target: credit_risk\n* Properties: twoclass\n* Features (20):\n - fct (14): credit_history, employment_duration, foreign_worker, housing, job, other_debtors,\n other_installment_plans, people_liable, personal_status_sex, property, purpose, savings, status,\n telephone\n - int (3): age, amount, duration\n - ord (3): installment_rate, number_credits, present_residence\n\nlearner = lrn(\"classif.xgboost\", nrounds = 100)\nprint(learner)\n\n\n* Model: -\n* Parameters: nrounds=100, nthread=1, verbose=0\n* Packages: mlr3, mlr3learners, xgboost\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric\n* Properties: hotstart_forward, importance, missings, multiclass, twoclass, weights\n\n\nWe now compare the feature types of the task and the supported feature types:\n\nunique(task$feature_types$type)\n\n[1] \"integer\" \"factor\" \"ordered\"\n\nlearner$feature_types\n\n[1] \"logical\" \"integer\" \"numeric\"\n\nsetdiff(task$feature_types$type, learner$feature_types)\n\n[1] \"factor\" \"ordered\"\n\n\nIn this example, we have to convert factors and ordered factors to numeric columns to apply the xgboost learner. Because xgboost is based on decision trees (at least in its default settings), it is perfectly fine to convert the ordered factors to integer. Unordered factors must still be encoded though." - }, - { - "objectID": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html#construct-pipeline", - "href": "gallery/2020-01-31-encode-factors-for-xgboost/2020-01-31-encode-factors-for-xgboost.html#construct-pipeline", - "title": "Encode Factor Levels for xgboost", - "section": "Construct Pipeline", - "text": "Construct Pipeline\nFinally, we construct a linear pipeline consisting of\n\nthe factor encoder fencoder,\nthe ordered factor converter ord_to_int, and\nthe xgboost base learner.\n\n\ngraph = fencoder %>>% ord_to_int %>>% learner\nprint(graph)\n\nGraph with 3 PipeOps:\n ID State sccssors prdcssors\n encode colapply \n colapply classif.xgboost encode\n classif.xgboost <> colapply\n\n\nThe pipeline is wrapped in a GraphLearner so that it behaves like a regular learner:\n\ngraph_learner = as_learner(graph)\n\nWe can now apply the new learner on the task, here with a 3-fold cross validation:\n\nrr = resample(task, graph_learner, rsmp(\"cv\", folds = 3))\nrr$aggregate()\n\nclassif.ce \n 0.2620435 \n\n\nSuccess! We augmented xgboost with handling of factors and ordered factors. If we combine this learner with a tuner from mlr3tuning, we get a universal and competitive learner." - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html", - "title": "mlr3tuning Tutorial - German Credit", - "section": "", - "text": "This is the second part of a serial of tutorials. The other parts of this series can be found here:\n\nPart I - Basics\nPart III - Pipelines\n\nWe will continue working with the German credit dataset. In Part I, we peeked into the dataset by using and comparing some learners with their default parameters. We will now see how to:\n\nTune hyperparameters for a given problem\nPerform nested resampling" - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#evaluation", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#evaluation", - "title": "mlr3tuning Tutorial - German Credit", - "section": "Evaluation", - "text": "Evaluation\nWe will evaluate all hyperparameter configurations using 10-fold cross-validation. We use a fixed train-test split, i.e. the same splits for each evaluation. Otherwise, some evaluation could get unusually “hard” splits, which would make comparisons unfair.\n\ncv10 = rsmp(\"cv\", folds = 10)\n\n# fix the train-test splits using the $instantiate() method\ncv10$instantiate(task)\n\n# have a look at the test set instances per fold\ncv10$instance\n\n row_id fold\n 1: 18 1\n 2: 19 1\n 3: 35 1\n 4: 38 1\n 5: 55 1\n --- \n 996: 973 10\n 997: 975 10\n 998: 981 10\n 999: 993 10\n1000: 998 10" - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#search-space-and-problem-definition", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#search-space-and-problem-definition", - "title": "mlr3tuning Tutorial - German Credit", - "section": "Search Space and Problem Definition", - "text": "Search Space and Problem Definition\nFirst, we need to decide what Learner we want to optimize. We will use LearnerClassifKKNN, the “kernelized” k-nearest neighbor classifier. We will use kknn as a normal kNN without weighting first (i.e., using the rectangular kernel):\n\nknn = lrn(\"classif.kknn\", predict_type = \"prob\", kernel = \"rectangular\")\n\nAs a next step, we decide what parameters we optimize over. Before that, though, we are interested in the parameter set on which we could tune:\n\nknn$param_set\n\n\n\n id class lower upper nlevels\n1: k ParamInt 1 Inf Inf\n2: distance ParamDbl 0 Inf Inf\n3: kernel ParamFct NA NA 10\n4: scale ParamLgl NA NA 2\n5: ykernel ParamUty NA NA Inf\n6: store_model ParamLgl NA NA 2\n\n\nWe first tune the k parameter (i.e. the number of nearest neighbors), between 3 to 20. Second, we tune the distance function, allowing L1 and L2 distances. To do so, we use the paradox package to define a search space (see the online vignette for a more complete introduction.\n\nsearch_space = ps(\n k = p_int(3, 20),\n distance = p_int(1, 2)\n)\n\nAs a next step, we define a TuningInstanceSingleCrit that represents the problem we are trying to optimize.\n\ninstance_grid = TuningInstanceSingleCrit$new(\n task = task,\n learner = knn,\n resampling = cv10,\n measure = msr(\"classif.ce\"),\n terminator = trm(\"none\"),\n search_space = search_space\n)" - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#grid-search", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#grid-search", - "title": "mlr3tuning Tutorial - German Credit", - "section": "Grid Search", - "text": "Grid Search\nAfter having set up a tuning instance, we can start tuning. Before that, we need a tuning strategy, though. A simple tuning method is to try all possible combinations of parameters: Grid Search. While it is very intuitive and simple, it is inefficient if the search space is large. For this simple use case, it suffices, though. We get the grid_search tuner via:\n\ntuner_grid = tnr(\"grid_search\", resolution = 18, batch_size = 36)\n\nTuning works by calling $optimize(). Note that the tuning procedure modifies our tuning instance (as usual for R6 class objects). The result can be found in the instance object. Before tuning it is empty:\n\ninstance_grid$result\n\nNULL\n\n\nNow, we tune:\n\ntuner_grid$optimize(instance_grid)\n\n k distance learner_param_vals x_domain classif.ce\n1: 7 1 0.25\n\n\n\n\n\n\n\n\nThe result is returned by $optimize() together with its performance. It can be also accessed with the $result slot:\n\ninstance_grid$result\n\n k distance learner_param_vals x_domain classif.ce\n1: 7 1 0.25\n\n\nWe can also look at the Archive of evaluated configurations:\n\nhead(as.data.table(instance_grid$archive))\n\n\n\n k distance classif.ce runtime_learners timestamp batch_nr warnings errors\n1: 3 1 0.273 0.470 2022-09-17 18:28:21 1 0 0\n2: 3 2 0.280 0.306 2022-09-17 18:28:21 1 0 0\n3: 4 1 0.290 0.445 2022-09-17 18:28:21 1 0 0\n4: 4 2 0.266 0.271 2022-09-17 18:28:21 1 0 0\n5: 5 1 0.268 0.483 2022-09-17 18:28:21 1 0 0\n6: 5 2 0.256 0.274 2022-09-17 18:28:21 1 0 0\n\n\nWe plot the performances depending on the sampled k and distance:\n\nggplot(as.data.table(instance_grid$archive),\n aes(x = k, y = classif.ce, color = as.factor(distance))) +\n geom_line() + geom_point(size = 3)\n\n\n\n\n\n\n\n\nOn average, the Euclidean distance (distance = 2) seems to work better. However, there is much randomness introduced by the resampling instance. So you, the reader, may see a different result, when you run the experiment yourself and set a different random seed. For k, we find that values between 7 and 13 perform well." - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#random-search-and-transformation", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#random-search-and-transformation", - "title": "mlr3tuning Tutorial - German Credit", - "section": "Random Search and Transformation", - "text": "Random Search and Transformation\nLet’s have a look at a larger search space. For example, we could tune all available parameters and limit k to large values (50). We also now tune the distance param continuously from 1 to 3 as a double and tune distance kernel and whether we scale the features.\nWe may find two problems when doing so:\nFirst, the resulting difference in performance between k = 3 and k = 4 is probably larger than the difference between k = 49 and k = 50. While 4 is 33% larger than 3, 50 is only 2 percent larger than 49. To account for this we will use a transformation function for k and optimize in log-space. We define the range for k from log(3) to log(50) and exponentiate in the transformation. Now, as k has become a double instead of an int (in the search space, before transformation), we round it in the extra_trafo.\n\nsearch_space_large = ps(\n k = p_dbl(log(3), log(50)),\n distance = p_dbl(1, 3),\n kernel = p_fct(c(\"rectangular\", \"gaussian\", \"rank\", \"optimal\")),\n scale = p_lgl(),\n .extra_trafo = function(x, param_set) {\n x$k = round(exp(x$k))\n x\n }\n)\n\nThe second problem is that grid search may (and often will) take a long time. For instance, trying out three different values for k, distance, kernel, and the two values for scale will take 54 evaluations. Because of this, we use a different search algorithm, namely the Random Search. We need to specify in the tuning instance a termination criterion. The criterion tells the search algorithm when to stop. Here, we will terminate after 36 evaluations:\n\ntuner_random = tnr(\"random_search\", batch_size = 36)\n\ninstance_random = TuningInstanceSingleCrit$new(\n task = task,\n learner = knn,\n resampling = cv10,\n measure = msr(\"classif.ce\"),\n terminator = trm(\"evals\", n_evals = 36),\n search_space = search_space_large\n)\n\n\ntuner_random$optimize(instance_random)\n\n k distance kernel scale learner_param_vals x_domain classif.ce\n1: 1.683743 1.985146 gaussian TRUE 0.254\n\n\n\n\n\n\n\n\nLike before, we can review the Archive. It includes the points before and after the transformation. The archive includes a column for each parameter the Tuner sampled on the search space (values before the transformation) and additional columns with prefix x_domain_* that refer to the parameters used by the learner (values after the transformation):\n\nas.data.table(instance_random$archive)\n\n\n\n\n\n\n\n\nLet’s now investigate the performance by parameters. This is especially easy using visualization:\n\nggplot(as.data.table(instance_random$archive),\n aes(x = x_domain_k, y = classif.ce, color = x_domain_scale)) +\n geom_point(size = 3) + geom_line()\n\n\n\n\n\n\n\n\nThe previous plot suggests that scale has a strong influence on performance. For the kernel, there does not seem to be a strong influence:\n\nggplot(as.data.table(instance_random$archive),\n aes(x = x_domain_k, y = classif.ce, color = x_domain_kernel)) +\n geom_point(size = 3) + geom_line()" - }, - { - "objectID": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#example-tuning-with-a-larger-budget", - "href": "gallery/2020-03-11-mlr3tuning-tutorial-german-credit/2020-03-11-mlr3tuning-tutorial-german-credit.html#example-tuning-with-a-larger-budget", - "title": "mlr3tuning Tutorial - German Credit", - "section": "Example: Tuning With A Larger Budget", - "text": "Example: Tuning With A Larger Budget\nIt is always interesting to look at what could have been. The following dataset contains an optimization run result with 3600 evaluations – more than above by a factor of 100:\n\n\n\n\n\n\n\n\n\n\nThe scale effect is just as visible as before with fewer data:\n\nggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = scale)) +\n geom_point(size = 2, alpha = 0.3)\n\n\n\n\n\n\n\n\nNow, there seems to be a visible pattern by kernel as well:\n\nggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = kernel)) +\n geom_point(size = 2, alpha = 0.3)\n\n\n\n\n\n\n\n\nIn fact, if we zoom in to (5, 40) \\(\\times\\) (0.23, 0.28) and do decrease smoothing we see that different kernels have their optimum at different values of k:\n\nggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = kernel,\n group = interaction(kernel, scale))) +\n geom_point(size = 2, alpha = 0.3) + geom_smooth() +\n xlim(5, 40) + ylim(0.23, 0.28)\n\n`geom_smooth()` using method = 'loess' and formula 'y ~ x'\n\n\n\n\n\n\n\n\n\nWhat about the distance parameter? If we select all results with k between 10 and 20 and plot distance and kernel we see an approximate relationship:\n\nggplot(perfdata[x_domain_k > 10 & x_domain_k < 20 & scale == TRUE],\n aes(x = distance, y = classif.ce, color = kernel)) +\n geom_point(size = 2) + geom_smooth()\n\n`geom_smooth()` using method = 'loess' and formula 'y ~ x'\n\n\n\n\n\n\n\n\n\nIn sum our observations are: The scale parameter is very influential, and scaling is beneficial. The distance type seems to be the least influential. There seems to be an interaction between ‘k’ and ‘kernel’." - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html", - "title": "Threshold Tuning for Classification Tasks", - "section": "", - "text": "Predicting probabilities in classification tasks allows us to adjust the probability thresholds required for assigning an observation to a certain class. This can lead to improved classification performance, especially for cases where we e.g. aim to balance off metrics such as false positive and false negative rates.\nThis is for example often done in ROC Analysis. The mlr3book also has a chapter on ROC Analysis) for the interested reader. This post does not focus on ROC analysis, but instead focusses on the general problem of adjusting classification thresholds for arbitrary metrics.\nThis post assumes some familiarity with the mlr3, and also the mlr3pipelines and mlr3tuning packages, as both are used during the post. The mlr3book contains more details on those two packages. This post is a more in-depth version of the article on threshold tuning in the mlr3book." - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#prerequisites", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#prerequisites", - "title": "Threshold Tuning for Classification Tasks", - "section": "Prerequisites", - "text": "Prerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#thresholds-a-short-intro", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#thresholds-a-short-intro", - "title": "Threshold Tuning for Classification Tasks", - "section": "Thresholds: A short intro", - "text": "Thresholds: A short intro\nIn order to understand thresholds, we will quickly showcase the effect of setting different thresholds:\nFirst we create a learner that predicts probabilities and use it to predict on holdout data, storing the prediction.\n\nlearner = lrn(\"classif.rpart\", predict_type = \"prob\")\nrr = resample(tsk(\"pima\"), learner, rsmp(\"holdout\"))\nprd = rr$prediction()\nprd\n\n for 256 observations:\n row_ids truth response prob.pos prob.neg\n 4 neg neg 0.1057692 0.8942308\n 6 neg neg 0.0200000 0.9800000\n 10 pos neg 0.1428571 0.8571429\n--- \n 764 neg neg 0.2777778 0.7222222\n 766 neg neg 0.0200000 0.9800000\n 767 pos pos 0.8000000 0.2000000\n\n\nIf we now look at the confusion matrix, the off-diagonal elements are errors made by our model (false positives and false negatives) while on-diagol ements are where our model predicted correctly.\n\n# Print confusion matrix\nprd$confusion\n\n truth\nresponse pos neg\n pos 53 27\n neg 37 139\n\n# Print False Positives and False Negatives\nprd$score(list(msr(\"classif.fp\"), msr(\"classif.fn\")))\n\nclassif.fp classif.fn \n 27 37 \n\n\nBy adjusting the classification threshold, in this case the probability required to predict the positive class, we can now trade off predicting more positive cases (first row) against predicting fewer negative cases (second row) or vice versa.\n\n# Lower threshold: More positives\nprd$set_threshold(0.25)$confusion\n\n truth\nresponse pos neg\n pos 78 71\n neg 12 95\n\n\n\n# Higher threshold: Fewer positives\nprd$set_threshold(0.75)$confusion\n\n truth\nresponse pos neg\n pos 52 20\n neg 38 146\n\n\nThis threshold value can now be adjusted optimally for a given measure, such as accuracy. How this can be done is discussed in the following section." - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#adjusting-thresholds-two-strategies", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#adjusting-thresholds-two-strategies", - "title": "Threshold Tuning for Classification Tasks", - "section": "Adjusting thresholds: Two strategies", - "text": "Adjusting thresholds: Two strategies\nCurrently mlr3pipelines offers two main strategies towards adjusting classification thresholds. We can either expose the thresholds as a hyperparameter of the Learner by using PipeOpThreshold. This allows us to tune the thresholds via an outside optimizer from mlr3tuning.\nAlternatively, we can also use PipeOpTuneThreshold which automatically tunes the threshold after each learner fit.\nIn this blog-post, we’ll go through both strategies." - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#pipeopthreshold", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#pipeopthreshold", - "title": "Threshold Tuning for Classification Tasks", - "section": "PipeOpThreshold", - "text": "PipeOpThreshold\nPipeOpThreshold can be put directly after a Learner.\nA simple example would be:\n\ngr = lrn(\"classif.rpart\", predict_type = \"prob\") %>>% po(\"threshold\")\nl = GraphLearner$new(gr)\n\nNote, that predict_type = “prob” is required for po(\"threshold\") to have any effect.\nThe thresholds are now exposed as a hyperparameter of the GraphLearner we created:\n\nl$param_set\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n classif.rpart.cp \n ParamDbl \n 0 \n 1 \n Inf \n \n \n classif.rpart.keep_model \n ParamLgl \n NA \n NA \n 2 \n \n \n classif.rpart.maxcompete \n ParamInt \n 0 \n Inf \n Inf \n \n \n classif.rpart.maxdepth \n ParamInt \n 1 \n 30 \n 30 \n \n \n classif.rpart.maxsurrogate \n ParamInt \n 0 \n Inf \n Inf \n \n \n classif.rpart.minbucket \n ParamInt \n 1 \n Inf \n Inf \n \n \n classif.rpart.minsplit \n ParamInt \n 1 \n Inf \n Inf \n \n \n classif.rpart.surrogatestyle \n ParamInt \n 0 \n 1 \n 2 \n \n \n classif.rpart.usesurrogate \n ParamInt \n 0 \n 2 \n 3 \n \n \n classif.rpart.xval \n ParamInt \n 0 \n Inf \n Inf \n \n \n threshold.thresholds \n ParamUty \n NA \n NA \n Inf \n \n\n\n\n\nWe can now tune those thresholds from the outside as follows:\nBefore tuning, we have to define which hyperparameters we want to tune over. In this example, we only tune over the thresholds parameter of the threshold PipeOp. you can easily imagine, that we can also jointly tune over additional hyperparameters, i.e. rpart’s cp parameter.\nAs the Task we aim to optimize for is a binary task, we can simply specify the threshold parameter:\n\nsearch_space = ps(\n threshold.thresholds = p_dbl(lower = 0, upper = 1)\n)\n\nWe now create a AutoTuner, which automatically tunes the supplied learner over the ParamSet we supplied above.\n\nat = auto_tuner(\n method = \"random_search\",\n learner = l,\n resampling = rsmp(\"cv\", folds = 3L),\n measure = msr(\"classif.ce\"),\n search_space = search_space,\n term_evals = 5L,\n)\n\nat$train(tsk(\"german_credit\"))\n\nFor multi-class Tasks, this is a little more complicated. We have to use a trafo to transform a set of ParamDbl into the desired format for threshold.thresholds: A named numeric vector containing the thresholds. This can be easily achieved via a trafo function:\n\nsearch_space = ps(\n versicolor = p_dbl(lower = 0, upper = 1),\n setosa = p_dbl(lower = 0, upper = 1),\n virginica = p_dbl(lower = 0, upper = 1),\n .extra_trafo = function(x, param_set) {\n list(threshold.thresholds = mlr3misc::map_dbl(x, identity))\n }\n)\n\nInside the .exta_trafo, we simply collect all set params into a named vector via map_dbl and store it in the threshold.thresholds slot expected by the learner.\nAgain, we create a AutoTuner, which automatically tunes the supplied learner over the ParamSet we supplied above.\n\nat_2 = auto_tuner(\n method = \"random_search\",\n learner = l,\n resampling = rsmp(\"cv\", folds = 3L),\n measure = msr(\"classif.ce\"),\n search_space = search_space,\n term_evals = 5L,\n)\n\nat_2$train(tsk(\"iris\"))\n\nOne drawback of this strategy is, that this requires us to fit a new model for each new threshold setting. While setting a threshold and computing performance is relatively cheap, fitting the learner is often more computationally demanding. A better strategy is therefore often to optimize the thresholds separately after each model fit." - }, - { - "objectID": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#pipeoptunethreshold", - "href": "gallery/2020-10-14-threshold-tuning/2020-10-14-threshold-tuning.html#pipeoptunethreshold", - "title": "Threshold Tuning for Classification Tasks", - "section": "PipeOpTuneThreshold", - "text": "PipeOpTuneThreshold\nPipeOpTuneThreshold on the other hand works together with PipeOpLearnerCV. It directly optimizes the cross-validated predictions made by this PipeOp.\nA simple example would be:\n\ngr = po(\"learner_cv\", lrn(\"classif.rpart\", predict_type = \"prob\")) %>>%\n po(\"tunethreshold\")\nl2 = GraphLearner$new(gr)\n\nNote, that predict_type = “prob” is required for po(\"tunethreshold\") to have any effect. Additionally, note that this time no threshold parameter is exposed, it is automatically tuned internally.\n\nl2$param_set\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n classif.rpart.resampling.method \n ParamFct \n NA \n NA \n 2 \n \n \n classif.rpart.resampling.folds \n ParamInt \n 2 \n Inf \n Inf \n \n \n classif.rpart.resampling.keep_response \n ParamLgl \n NA \n NA \n 2 \n \n \n classif.rpart.cp \n ParamDbl \n 0 \n 1 \n Inf \n \n \n classif.rpart.keep_model \n ParamLgl \n NA \n NA \n 2 \n \n \n classif.rpart.maxcompete \n ParamInt \n 0 \n Inf \n Inf \n \n \n classif.rpart.maxdepth \n ParamInt \n 1 \n 30 \n 30 \n \n \n classif.rpart.maxsurrogate \n ParamInt \n 0 \n Inf \n Inf \n \n \n classif.rpart.minbucket \n ParamInt \n 1 \n Inf \n Inf \n \n \n classif.rpart.minsplit \n ParamInt \n 1 \n Inf \n Inf \n \n \n classif.rpart.surrogatestyle \n ParamInt \n 0 \n 1 \n 2 \n \n \n classif.rpart.usesurrogate \n ParamInt \n 0 \n 2 \n 3 \n \n \n classif.rpart.xval \n ParamInt \n 0 \n Inf \n Inf \n \n \n classif.rpart.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n tunethreshold.measure \n ParamUty \n NA \n NA \n Inf \n \n \n tunethreshold.optimizer \n ParamUty \n NA \n NA \n Inf \n \n \n tunethreshold.log_level \n ParamUty \n NA \n NA \n Inf \n \n\n\n\n\nIf we now use the GraphLearner, it automatically adjusts the thresholds during prediction.\nNote that we can set ResamplingInsample as a resampling strategy for PipeOpLearnerCV in order to evaluate predictions on the “training” data. This is generally not advised, as it might lead to over-fitting on the thresholds but can significantly reduce runtime.\nFinally, we can compare no threshold tuning to the tunethreshold approach:\n\nComparison of the approaches\n\nbmr = benchmark(benchmark_grid(\n learners = list(no_tuning = lrn(\"classif.rpart\"), internal = l2),\n tasks = tsk(\"german_credit\"),\n rsmp(\"cv\", folds = 3L)\n))\n\n\nbmr$aggregate(list(msr(\"classif.ce\"), msr(\"classif.fnr\")))\n\n\n\n\n\n \n \n nr \n task_id \n learner_id \n resampling_id \n iters \n classif.ce \n classif.fnr \n \n \n\n \n 1 \n german_credit \n classif.rpart \n cv \n 3 \n 0.2760095 \n 0.1272398 \n \n \n 2 \n german_credit \n classif.rpart.tunethreshold \n cv \n 3 \n 0.2879916 \n 0.0448533" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "", - "text": "This is the third part of a serial of use cases with the German credit dataset. The other parts of this series can be found here:\n\nPart I - Basics\nPart II - Tuning\n\nIn this tutorial, we continue working with the German credit dataset. We already used different Learners on it and tried to optimize their hyperparameters. Now we will do four additional things:\n\nWe preprocess the data as an integrated step of the model fitting process\nWe tune the associated preprocessing parameters\nWe stack multiple Learners in an ensemble model\nWe discuss some techniques that make Learners able to tackle challenging datasets that they could not handle otherwise (we are going to outline what challenging means in particular later on)" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#prerequisites", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#prerequisites", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Prerequisites", - "text": "Prerequisites\nFirst, load the packages we are going to use:\n\nlibrary(\"mlr3verse\")\nlibrary(\"data.table\")\nlibrary(\"ggplot2\")\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nWe again use the German credit dataset, but will restrict ourselves to the factorial features. To make things interesting or to make it a bit harder for our Learners, we introduce missing values in the dataset:\n\ntask = tsk(\"german_credit\")\ncredit_full = task$data()\ncredit = credit_full[, sapply(credit_full, FUN = is.factor), with = FALSE]\n\n# sample values to NA\ncredit = credit[, lapply(.SD, function(x) {\n x[sample(c(TRUE, NA), length(x), replace = TRUE, prob = c(.9, .1))]\n})]\ncredit$credit_risk = credit_full$credit_risk\ntask = TaskClassif$new(\"GermanCredit\", credit, \"credit_risk\")\n\nWe instantiate a Resampling instance for this Task to be able to compare resampling performance:\n\ncv10 = rsmp(\"cv\")$instantiate(task)\n\nWe also might want to use multiple cores to reduce long run times of tuning runs.\n\nfuture::plan(\"multiprocess\")" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#intro", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#intro", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Intro", - "text": "Intro\nIn this use case, we will take a look at composite machine learning algorithms that may incorporate data preprocessing or the combination of multiple Learners (“ensemble methods”).\nWe use the mlr3pipelines package that enables us to chain PipeOps into data flow Graphs.\nAvailable PipeOps are listed in the mlr_pipeops dictionary:\n\nmlr_pipeops\n\n with 64 stored values\nKeys: boxcox, branch, chunk, classbalancing, classifavg, classweights, colapply, collapsefactors, colroles,\n copy, datefeatures, encode, encodeimpact, encodelmer, featureunion, filter, fixfactors, histbin, ica,\n imputeconstant, imputehist, imputelearner, imputemean, imputemedian, imputemode, imputeoor, imputesample,\n kernelpca, learner, learner_cv, missind, modelmatrix, multiplicityexply, multiplicityimply, mutate, nmf,\n nop, ovrsplit, ovrunite, pca, proxy, quantilebin, randomprojection, randomresponse, regravg,\n removeconstants, renamecolumns, replicate, scale, scalemaxabs, scalerange, select, smote, spatialsign,\n subsample, targetinvert, targetmutate, targettrafoscalerange, textvectorizer, threshold, tunethreshold,\n unbranch, vtreat, yeojohnson" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#missing-value-imputation", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#missing-value-imputation", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Missing Value Imputation", - "text": "Missing Value Imputation\nWe have just introduced missing values into our data. While some Learners can deal with missing value, many cannot. Trying to train a random forest fails because of this:\n\nranger = lrn(\"classif.ranger\")\nranger$train(task)\n\nError: Task 'GermanCredit' has missing values in column(s) 'credit_history', 'employment_duration', 'foreign_worker', 'housing', 'installment_rate', 'job', 'number_credits', 'other_debtors', 'other_installment_plans', 'people_liable', 'personal_status_sex', 'present_residence', 'property', 'purpose', 'savings', 'status', 'telephone', but learner 'classif.ranger' does not support this\n\n\nWe can perform imputation of missing values using a PipeOp. To find out which imputation PipeOps are available, we do the following:\n\nmlr_pipeops$keys(\"^impute\")\n\n[1] \"imputeconstant\" \"imputehist\" \"imputelearner\" \"imputemean\" \"imputemedian\" \"imputemode\" \n[7] \"imputeoor\" \"imputesample\" \n\n\nWe choose to impute factorial features using a new level (via PipeOpImputeOOR). Let’s use the PipeOp itself to create an imputed Task. This shows us how the PipeOp actually works:\n\nimputer = po(\"imputeoor\")\ntask_imputed = imputer$train(list(task))[[1]]\ntask_imputed$missings()\nhead(task_imputed$data())\n\n\n\n\n\n\n\n\nWe do not only need complete data during training but also during prediction. Using the same imputation heuristic for both is the most consistent strategy. This way the imputation strategy can, in fact, be seen as a part of the complete learner (which could be tuned).\nIf we used the imputed Task for Resampling, we would leak information from the test set into the training set. Therefore, it is mandatory to attach the imputation operator to the Learner itself, creating a GraphLearner:\n\ngraph_learner_ranger = as_learner(po(\"imputeoor\") %>>% ranger)\n\ngraph_learner_ranger$train(task)\n\nThis GraphLearner can be used for resampling – like an ordinary Learner:\n\nrr = resample(task, learner = graph_learner_ranger, resampling = cv10)\nrr$aggregate()\n\nclassif.ce \n 0.29" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#feature-filtering", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#feature-filtering", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Feature Filtering", - "text": "Feature Filtering\nTypically, sparse models, i.e. having models with few(er) features, are desirable. This is due to a variety of reasons, e.g., enhanced interpretability or decreased costs of acquiring data. Furthermore, sparse models may actually be associated with increased performance (especially if overfitting is anticipated). We can use feature filter to only keep features with the highest information. Filters are implemented in the mlr3filters package and listed in the following dictionary:\n\nmlr_filters\n\n with 20 stored values\nKeys: anova, auc, carscore, cmim, correlation, disr, find_correlation, importance, information_gain, jmi,\n jmim, kruskal_test, mim, mrmr, njmim, performance, permutation, relief, selected_features, variance\n\n\nWe apply the FilterMIM (mutual information maximization) Filter as implemented in the praznik package. This Filter allows for the selection of the top-k features of best mutual information.\n\nfilter = flt(\"mim\")\nfilter$calculate(task_imputed)$scores\n\n status credit_history savings purpose property \n 1.0000 0.9375 0.8750 0.8125 0.7500 \n housing employment_duration other_installment_plans personal_status_sex other_debtors \n 0.6875 0.6250 0.5625 0.5000 0.4375 \n installment_rate foreign_worker job number_credits telephone \n 0.3750 0.3125 0.2500 0.1875 0.1250 \n present_residence people_liable \n 0.0625 0.0000 \n\n\nMaking use of this Filter, you may wonder at which costs the reduction of the feature space comes. We can investigate the trade-off between features and performance by tuning. We incorporate our filtering strategy into the pipeline using PipeOpFilter. Like before, we need to perform imputation as the Filter also relies on complete data:\n\nfpipe = po(\"imputeoor\") %>>% po(\"filter\", flt(\"mim\"), filter.nfeat = 3)\nfpipe$train(task)[[1]]$head()\n\n credit_risk credit_history savings\n1: good all credits at this bank paid back duly ... >= 1000 DM\n2: bad no credits taken/all credits paid back duly unknown/no savings account\n3: good all credits at this bank paid back duly unknown/no savings account\n4: good no credits taken/all credits paid back duly unknown/no savings account\n5: bad existing credits paid back duly till now unknown/no savings account\n6: good .MISSING ... >= 1000 DM\n status\n1: no checking account\n2: ... < 0 DM\n3: ... >= 200 DM / salary for at least 1 year\n4: no checking account\n5: no checking account\n6: ... >= 200 DM / salary for at least 1 year\n\n\nWe can now tune over the mim.filter.nfeat parameter. It steers how many features are kept by the Filter and eventually used by the learner:\n\nsearch_space = ps(\n mim.filter.nfeat = p_int(lower = 1, upper = length(task$feature_names))\n)\n\nThe problem is one-dimensional (i.e. only one parameter is tuned). Thus, we make use of a grid search. For higher dimensions, strategies like random search are more appropriate. The tuning procedure may take some time:\n\ninstance = tune(\n method = \"grid_search\",\n task,\n learner = fpipe %>>% lrn(\"classif.ranger\"),\n resampling = cv10,\n measure = msr(\"classif.ce\"),\n search_space = search_space)\n\n\n\n\n\n\n\nWe can plot the performance against the number of features. If we do so, we see the possible trade-off between sparsity and predictive performance:\n\nautoplot(instance, type = \"marginal\")" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#stacking", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#stacking", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Stacking", - "text": "Stacking\nWe want to build a model that is based on the predictions of other Learners. This means that we are in the state that we need predictions already during training. This is a very specific case that is luckily handled by PipeOpLearnerCV. PipeOpLearnerCV performs cross-validation during the training phase and returns the cross-validated predictions. We use \"prob\" predictions because they carry more information than response prediction:\n\ngraph_stack = po(\"imputeoor\") %>>%\n gunion(list(\n po(\"learner_cv\", lrn(\"classif.ranger\", predict_type = \"prob\")),\n po(\"learner_cv\", lrn(\"classif.kknn\", predict_type = \"prob\")))) %>>%\n po(\"featureunion\") %>>% lrn(\"classif.log_reg\")\n\nWe built a pretty complex Graph already. Therefore, we plot it:\n\ngraph_stack$plot(html = TRUE)\n\n\n\n\n\nWe now compare the performance of the stacked learner to the performance of the individual Learners:\n\ngrid = benchmark_grid(\n task = task,\n learner = list(\n graph_stack,\n as_learner(po(\"imputeoor\") %>>% lrn(\"classif.ranger\")),\n as_learner(po(\"imputeoor\") %>>% lrn(\"classif.kknn\")),\n as_learner(po(\"imputeoor\") %>>% lrn(\"classif.log_reg\"))),\n resampling = cv10)\n\nbmr = benchmark(grid)\n\n\n\n\n\n\n\n\n\n learner_id classif.ce\n1: imputeoor.classif.ranger.classif.kknn.featureunion.classif.log_reg 0.282\n2: imputeoor.classif.ranger 0.291\n3: imputeoor.classif.kknn 0.300\n4: imputeoor.classif.log_reg 0.283\n\n\nIf we train the stacked learner and look into the final Learner (the logistic regression), we can see how “important” each Learner of the stacked learner is:\n\ngraph_stack$train(task)\n\n$classif.log_reg.output\nNULL\n\nsummary(graph_stack$pipeops$classif.log_reg$state$model)\n\n\nCall:\nstats::glm(formula = task$formula(), family = \"binomial\", data = data, \n model = FALSE)\n\nDeviance Residuals: \n Min 1Q Median 3Q Max \n-2.3441 -0.9954 0.5294 0.8024 1.9788 \n\nCoefficients: (2 not defined because of singularities)\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) -3.3236 0.3542 -9.385 < 2e-16 ***\nclassif.ranger.prob.good 5.3948 0.5667 9.520 < 2e-16 ***\nclassif.ranger.prob.bad NA NA NA NA \nclassif.kknn.prob.good 0.8557 0.3169 2.700 0.00692 ** \nclassif.kknn.prob.bad NA NA NA NA \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 1221.7 on 999 degrees of freedom\nResidual deviance: 1054.7 on 997 degrees of freedom\nAIC: 1060.7\n\nNumber of Fisher Scoring iterations: 4\n\n\nThe random forest has a higher contribution." - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#robustify-preventing-new-prediction-factor-levels-and-other-problems", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#robustify-preventing-new-prediction-factor-levels-and-other-problems", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Robustify: Preventing new Prediction Factor Levels and other Problems", - "text": "Robustify: Preventing new Prediction Factor Levels and other Problems\nWe now shift the context, using the complete German credit dataset:\n\ntask = tsk(\"german_credit\")\n\nThere is a potential practical problem for both, small data sets and data sets with covariates having many factor levels: It may occur that not all possible factor levels have been used by the Learner during training. This happens because these rare instances are simply not sampled. The prediction then may fail because the Learner does not know how to handle unseen factor levels:\n\ntask_unseen = task$clone()$filter(1:30)\nlearner_logreg = lrn(\"classif.log_reg\")\nlearner_logreg$train(task_unseen)\nlearner_logreg$predict(task)\n\nError in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels): factor job has new levels unemployed/unskilled - non-resident\n\n\nNot only logistic regression but also many other Learners cannot handle new levels during prediction. Thus, we use PipeOpFixFactors to prevent that. PipeOpFixFactors introduces NA values for unseen levels. This means that we may need to impute afterwards. To solve this issue we can use PipeOpImputeSample, but with affect_columns set to only factorial features.\nAnother observation is that all-constant features may also be a problem:\n\ntask_constant = task$clone()$filter(1:2)\nlearner_logreg = lrn(\"classif.log_reg\")\nlearner_logreg$train(task_constant)\n\nError in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]): contrasts can be applied only to factors with 2 or more levels\n\n\nThis can be fixed using PipeOpRemoveConstants.\nBoth, handling unseen levels and all-constant features can be handled simultaneously using the following Graph:\n\nrobustify = po(\"fixfactors\") %>>%\n po(\"removeconstants\") %>>%\n po(\"imputesample\", affect_columns = selector_type(c(\"ordered\", \"factor\")))\n\nrobustify$plot(html = TRUE)\n\n\n\n\n\nThis robust learner works even in very pathological conditions:\n\ngraph_learner_robustify = as_learner(robustify %>>% learner_logreg)\n\ngraph_learner_robustify$train(task_constant)\ngraph_learner_robustify$predict(task)\n\n for 1000 observations:\n row_ids truth response\n 1 good good\n 2 bad bad\n 3 good good\n--- \n 998 good bad\n 999 bad bad\n 1000 good bad" - }, - { - "objectID": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#your-ideas", - "href": "gallery/2020-03-11-mlr3pipelines-tutorial-german-credit/2020-03-11-mlr3pipelines-tutorial-german-credit.html#your-ideas", - "title": "mlr3pipelines Tutorial - German Credit", - "section": "Your Ideas", - "text": "Your Ideas\nThere are various possibilities for preprocessing with PipeOps. You can try different methods for preprocessing and training. Feel free to discover this variety by yourself! Here are only a few hints that help when working with PipeOps:\n\nIt is not allowed to have two PipeOps with the same ID in a Graph\n\nInitialize a PipeOp with po(\"...\", id = \"xyz\") to change its ID on construction\n\nIf you build large Graphs involving complicated optimizations, like many \"learner_cv\", they may need a long time to train\nUse the affect_columns parameter if you want a PipeOp to only operate on part of the data\nUse po(\"select\") if you want to remove certain columns (possibly only along a single branch of multiple parallel branches). Both take selector_xxx() arguments, e.g. selector_type(\"integer\")\nYou may get the best performance if you actually inspect the features and see what kind of transformations work best for them (know your data!)\nSee what PipeOps are available by inspecting mlr_pipeops$keys(), and get help about them using ?mlr_pipeops_xxx" - }, - { - "objectID": "gallery/2020-04-18-regression-chains/regression-chains.html", - "href": "gallery/2020-04-18-regression-chains/regression-chains.html", - "title": "Regression Chains", - "section": "", - "text": "In this tutorial we demonstrate how to use mlr3pipelines to handle multi-target regression by arranging regression models as a chain, i.e., creating a linear sequence of regression models.\n\nRegression Chains\nIn a simple regression chain, regression models are arranged in a linear sequence. Here, the first model will use the input to predict a single output and the second model will use the input and the prediction output of the first model to make its own prediction and so on. For more details, see e.g. Spyromitros-Xioufis et al. (2016).\n\n\nBefore you start\nThe following sections describe an approach towards working with tasks that have multiple targets. E.g., in the example below, we have three target variables \\(y_{1}\\) to \\(y_{3}\\). This type of Task can be created via the mlr3multioutput package (currently under development) in the future. mlr3multioutput will also offer simple chaining approaches as pre-built pipelines (so called ppls). The current goal of this post is to show how such modeling steps can be written as a relatively small amount of pipeline steps and how such steps can be put together. Writing pipelines with such steps allows for great flexibility in modeling more complicated scenarios such as the ones described below.\n\n\nPrerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nLoading required package: mlr3\n\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\n\n\n\nData\nIn the following, we rely on some toy data. We simulate 100 responses to three target variables, \\(y_{1}\\), \\(y_{2}\\), and \\(y_{3}\\) following a multivariate normal distribution with a mean and covariance matrix of:\n\nlibrary(data.table)\nlibrary(mvtnorm)\nset.seed(2409)\nn = 100\n(mean = c(y1 = 1, y2 = 2, y3 = 3))\n\ny1 y2 y3 \n 1 2 3 \n\n(sigma = matrix(c(1, -0.5, 0.25, -0.5, 1, -0.25, 0.25, -0.25, 1),\n nrow = 3, ncol = 3, byrow = TRUE))\n\n [,1] [,2] [,3]\n[1,] 1.00 -0.50 0.25\n[2,] -0.50 1.00 -0.25\n[3,] 0.25 -0.25 1.00\n\nY = rmvnorm(n, mean = mean, sigma = sigma)\n\nThe feature variables \\(x_{1}\\), and \\(x_{2}\\) are simulated as follows: \\(x_{1}\\) is simply given by \\(y_{1}\\) and an independent normally distributed error term and \\(x_{2}\\) is given by \\(y_{2}\\) and an independent normally distributed error term.\n\nx1 = Y[, 1] + rnorm(n, sd = 0.1)\nx2 = Y[, 2] + rnorm(n, sd = 0.1)\n\nThe final data is given as:\n\ndata = as.data.table(cbind(Y, x1, x2))\nstr(data)\n\nClasses 'data.table' and 'data.frame': 100 obs. of 5 variables:\n $ y1: num 0.681 1.836 0.355 1.783 0.974 ...\n $ y2: num 2.33 1.735 3.126 0.691 1.573 ...\n $ y3: num 3.19 3.14 2.74 4.31 2.77 ...\n $ x1: num 0.788 1.754 0.174 1.844 1.05 ...\n $ x2: num 2.336 1.665 2.967 0.651 1.634 ...\n - attr(*, \".internal.selfref\")= \n\n\nThis simulates a situation where we have multiple target variables that are correlated with each other, such that predicting them along with each other can improve the resulting prediction model. As a real-world example for such a situation, consider e.g. hospital data, where time spent in the ICU (not known a priori) heavily influences the cost incurred by a patient’s treatment.\n\n\n3D Visualization of the Data\nIf you feel confident to already have a good feeling of the data, feel free to skip this section. If not, you can use the rgl package to play around with the following four 3D plots with either the feature variables or \\(y_{1}\\) and \\(y_{2}\\) on the x- and y-axis and the target variables on the respective z-axes:\n\nlibrary(rgl)\ncolfun = colorRampPalette(c(\"#161B1D\", \"#ADD8E6\"))\n\n\nsetorder(data, y1)\nplot3d(data$x1, data$x2, data$y1,\n xlab = \"x1\", ylab = \"x2\", zlab = \"y1\",\n type = \"s\", radius = 0.1, col = colfun(n))\n\n\nsetorder(data, y2)\nplot3d(data$x1, data$x2, data$y2,\n xlab = \"x1\", ylab = \"x2\", zlab = \"y2\",\n type = \"s\", radius = 0.1, col = colfun(n))\n\n\nsetorder(data, y3)\nplot3d(data$x1, data$x2, data$y3,\n xlab = \"x1\", ylab = \"x2\", zlab = \"y3\",\n type = \"s\", radius = 0.1, col = colfun(n))\n\n\nsetorder(data, y3)\nplot3d(data$y1, data$y2, data$y3,\n xlab = \"y1\", ylab = \"y2\", zlab = \"y3\",\n type = \"s\", radius = 0.1, col = colfun(n))\n\n\n\nBuilding the Pipeline\nIn our regression chain, the first model will predict \\(y_{1}\\). Therefore, we initialize our Task with respect to this target:\n\ntask = as_task_regr(data, id = \"multiregression\", target = \"y1\")\n\nAs Learners we will use simple linear regression models. Our pipeline building the regression chain then has to do the following:\n\nUse the input to predict \\(y_{1}\\) within the first learner (i.e., \\(y_{1} \\sim x_{1} + x_{2}\\)).\nCombine the input with the prediction of \\(y_{1}\\), \\(\\hat{y_{1}}\\) and use this to predict \\(y_{2}\\) within the second learner (i.e., \\(y_{2} \\sim x_{1} + x_{2} + \\hat{y_{1}}\\)).\nCombine the input with the prediction of \\(y_{2}\\) and use this to predict \\(y_{3}\\) within the final third learner (i.e., \\(y_{3} \\sim x_{1} + x_{2} + \\hat{y_{1}} + \\hat{y_{2}}\\)).\n\nTo combine predictions of a Learner with the previous input, we rely on PipeOpLearnerCV and PipeOpNOP arranged in parallel via gunion() combined via PipeOpFeatureUnion. To drop the respective remaining target variables as features, we rely on PipeOpColRoles. The first step of predicting \\(y_{1}\\) looks like the following:\n\nstep1 = po(\"copy\", outnum = 2, id = \"copy1\") %>>%\n gunion(list(\n po(\"colroles\", id = \"drop_y2_y3\",\n new_role = list(y2 = character(), y3 = character())) %>>%\n po(\"learner_cv\", learner = lrn(\"regr.lm\"), id = \"y1_learner\"),\n po(\"nop\", id = \"nop1\")\n )) %>>%\n po(\"featureunion\", id = \"union1\")\nstep1$plot(html = TRUE)\n\n\n\n\n\nTraining using the input Task, shows us how the output and the $state look like:\n\nstep1_out = step1$train(task)[[1]]\nstep1_out\n\n (100 x 6)\n* Target: y1\n* Properties: -\n* Features (5):\n - dbl (5): x1, x2, y1_learner.response, y2, y3\n\nstep1$state\n\n$copy1\nlist()\n\n$drop_y2_y3\n$drop_y2_y3$dt_columns\n[1] \"x1\" \"x2\" \"y2\" \"y3\"\n\n$drop_y2_y3$affected_cols\n[1] \"x1\" \"x2\" \"y2\" \"y3\"\n\n$drop_y2_y3$intasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n3: y2 numeric\n4: y3 numeric\n\n$drop_y2_y3$outtasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n\n$drop_y2_y3$outtaskshell\nEmpty data.table (0 rows and 3 cols): y1,x1,x2\n\n\n$y1_learner\n$y1_learner$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n(Intercept) x1 x2 \n -0.03762 0.99851 0.01364 \n\n\n$y1_learner$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$y1_learner$train_time\n[1] 0.002\n\n$y1_learner$param_vals\nnamed list()\n\n$y1_learner$task_hash\n[1] \"36ef60ec8dfba260\"\n\n$y1_learner$data_prototype\nEmpty data.table (0 rows and 3 cols): y1,x1,x2\n\n$y1_learner$task_prototype\nEmpty data.table (0 rows and 3 cols): y1,x1,x2\n\n$y1_learner$mlr3_version\n[1] '0.14.0'\n\n$y1_learner$train_task\n (100 x 3)\n* Target: y1\n* Properties: -\n* Features (2):\n - dbl (2): x1, x2\n\n$y1_learner$affected_cols\n[1] \"x1\" \"x2\"\n\n$y1_learner$intasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n\n$y1_learner$outtasklayout\n id type\n1: y1_learner.response numeric\n\n$y1_learner$outtaskshell\nEmpty data.table (0 rows and 2 cols): y1,y1_learner.response\n\n\n$nop1\nlist()\n\n$union1\nlist()\n\n\nWithin the second step we then have to define \\(y_{2}\\) as the new target. This can be done using PipeOpUpdateTarget (note that PipeOpUpdateTarget currently is not exported but will be in a future version). By default, PipeOpUpdateTarget drops the original target from the feature set, here \\(y_{1}\\).\n\nmlr_pipeops$add(\"update_target\", mlr3pipelines:::PipeOpUpdateTarget)\n\n\nstep2 = po(\"update_target\", id = \"y2_target\",\n new_target_name = \"y2\") %>>%\n po(\"copy\", outnum = 2, id = \"copy2\") %>>%\n gunion(list(\n po(\"colroles\", id = \"drop_y3\",\n new_role = list(y3 = character())) %>>%\n po(\"learner_cv\", learner = lrn(\"regr.lm\"), id = \"y2_learner\"),\n po(\"nop\", id = \"nop2\")\n )) %>>%\n po(\"featureunion\", id = \"union2\")\n\nAgain, we can train to see how the output and $state look like, but now using the output of step1 as the input:\n\nstep2_out = step2$train(step1_out)[[1]]\nstep2_out\n\n (100 x 6)\n* Target: y2\n* Properties: -\n* Features (5):\n - dbl (5): x1, x2, y1_learner.response, y2_learner.response, y3\n\nstep2$state\n\n$y2_target\nlist()\n\n$copy2\nlist()\n\n$drop_y3\n$drop_y3$dt_columns\n[1] \"x1\" \"x2\" \"y1_learner.response\" \"y3\" \n\n$drop_y3$affected_cols\n[1] \"y1_learner.response\" \"x1\" \"x2\" \"y3\" \n\n$drop_y3$intasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n3: y1_learner.response numeric\n4: y3 numeric\n\n$drop_y3$outtasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n3: y1_learner.response numeric\n\n$drop_y3$outtaskshell\nEmpty data.table (0 rows and 4 cols): y2,y1_learner.response,x1,x2\n\n\n$y2_learner\n$y2_learner$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n (Intercept) y1_learner.response x1 x2 \n 0.07135 0.22773 -0.25186 0.97877 \n\n\n$y2_learner$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$y2_learner$train_time\n[1] 0.003\n\n$y2_learner$param_vals\nnamed list()\n\n$y2_learner$task_hash\n[1] \"69223401fddacf96\"\n\n$y2_learner$data_prototype\nEmpty data.table (0 rows and 4 cols): y2,y1_learner.response,x1,x2\n\n$y2_learner$task_prototype\nEmpty data.table (0 rows and 4 cols): y2,y1_learner.response,x1,x2\n\n$y2_learner$mlr3_version\n[1] '0.14.0'\n\n$y2_learner$train_task\n (100 x 4)\n* Target: y2\n* Properties: -\n* Features (3):\n - dbl (3): x1, x2, y1_learner.response\n\n$y2_learner$affected_cols\n[1] \"y1_learner.response\" \"x1\" \"x2\" \n\n$y2_learner$intasklayout\n id type\n1: x1 numeric\n2: x2 numeric\n3: y1_learner.response numeric\n\n$y2_learner$outtasklayout\n id type\n1: y2_learner.response numeric\n\n$y2_learner$outtaskshell\nEmpty data.table (0 rows and 2 cols): y2,y2_learner.response\n\n\n$nop2\nlist()\n\n$union2\nlist()\n\n\nIn the final third step we define \\(y_{3}\\) as the new target (again, PipeOpUpdateTarget drops the previous original target from the feature set, here \\(y_{2}\\)):\n\nstep3 = po(\"update_target\", id = \"y3_target\",\n new_target_name = \"y3\") %>>%\n po(\"learner\", learner = lrn(\"regr.lm\"), id = \"y3_learner\")\n\nUsing the output of step2 as input:\n\nstep3_out = step3$train(step2_out)[[1]]\nstep3_out\n\nNULL\n\nstep3$state\n\n$y3_target\nlist()\n\n$y3_learner\n$y3_learner$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n (Intercept) y2_learner.response y1_learner.response x1 x2 \n 2.6445 0.8155 3.8776 -3.5217 -0.7304 \n\n\n$y3_learner$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$y3_learner$train_time\n[1] 0.004\n\n$y3_learner$param_vals\nnamed list()\n\n$y3_learner$task_hash\n[1] \"9ffb08b85dd193ec\"\n\n$y3_learner$data_prototype\nEmpty data.table (0 rows and 5 cols): y3,y2_learner.response,y1_learner.response,x1,x2\n\n$y3_learner$task_prototype\nEmpty data.table (0 rows and 5 cols): y3,y2_learner.response,y1_learner.response,x1,x2\n\n$y3_learner$mlr3_version\n[1] '0.14.0'\n\n$y3_learner$train_task\n (100 x 5)\n* Target: y3\n* Properties: -\n* Features (4):\n - dbl (4): x1, x2, y1_learner.response, y2_learner.response\n\n\nThe complete pipeline, more precisely Graph, looks like the following:\n\ngraph = step1 %>>% step2 %>>% step3\ngraph$plot(html = TRUE)\n\n\n\n\n\n\n\nEvaluating the Pipeline\nBy wrapping our Graph in a GraphLearner, we can perform 3-fold cross-validation and get an estimated average of the root-mean-square error (of course, in a real world setting splitting the data in a training and test set should have been done):\n\nlearner = as_learner(graph)\nrr = resample(task, learner, rsmp(\"cv\", folds = 3))\nrr$aggregate(msr(\"regr.mse\"))\n\n regr.mse \n0.7265587 \n\n\n\n\nPredicting with the Pipeline\nFor completeness, we also show how a prediction step without having any target variable data available would look like:\n\ndata_predict = as.data.table(cbind(x1, x2, y1 = NA, y2 = NA, y3 = NA))\nlearner$train(task)\nlearner$predict_newdata(data_predict)\n\n for 100 observations:\n row_ids truth response\n 1 NA 3.116960\n 2 NA 3.327345\n 3 NA 3.010821\n--- \n 98 NA 3.462541\n 99 NA 3.020585\n 100 NA 3.664326\n\n\nNote that we have to initialize the Task with \\(y_{1}\\) as the target but the pipeline will automatically predict \\(y_{3}\\) in the final step as our final target, which was our ultimate goal here.\n\n\n\n\n\nReferences\n\nSpyromitros-Xioufis, Eleftherios, Grigorios Tsoumakas, William Groves, and Ioannis Vlahavas. 2016. “Multi-Target Regression via Input Space Expansion: Treating Targets as Inputs.” Machine Learning 104 (1): 55–98. https://doi.org/10.1007/s10994-016-5546-z." - }, - { - "objectID": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html", - "href": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html", - "title": "A Pipeline for the Titanic Data Set - Basics", - "section": "", - "text": "We load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners. The data is part of the mlr3data package.\n\nlibrary(mlr3verse)\nlibrary(mlr3learners)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\n\nThe titanic data is very interesting to analyze, even though it is part of many tutorials and showcases. This is because it requires many steps often required in real-world applications of machine learning techniques, such as missing value imputation, handling factors and others.\nFollowing features are illustrated in this use case section:\n\nSummarizing the data set\nVisualizing data\nSplitting data into train and test data sets\nDefining a task and a learner" - }, - { - "objectID": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html#exploratory-data-analysis", - "href": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html#exploratory-data-analysis", - "title": "A Pipeline for the Titanic Data Set - Basics", - "section": "Exploratory Data Analysis", - "text": "Exploratory Data Analysis\nWith the dataset, we get an explanation of the meanings of the different variables:\n\n\n\nVariables\nDescription\n\n\n\n\nsurvived\nSurvival\n\n\nname\nName\n\n\nage\nAge\n\n\nsex\nSex\n\n\nsib_sp\nNumber of siblings / spouses aboard\n\n\nparch\nNumber of parents / children aboard\n\n\nfare\nAmount paid for the ticket\n\n\npc_class\nPassenger class\n\n\nembarked\nPort of embarkation\n\n\nticket\nTicket number\n\n\ncabin\nCabin\n\n\n\nWe can use the skimr package in order to get a first overview of the data:\n\ndata(\"titanic\", package = \"mlr3data\")\n\nskimr::skim(titanic)\n\n\nData summary\n\n\nName\ntitanic\n\n\nNumber of rows\n1309\n\n\nNumber of columns\n11\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\ncharacter\n3\n\n\nfactor\n4\n\n\nnumeric\n4\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: character\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmin\nmax\nempty\nn_unique\nwhitespace\n\n\n\n\nname\n0\n1.00\n12\n82\n0\n1307\n0\n\n\nticket\n0\n1.00\n3\n18\n0\n929\n0\n\n\ncabin\n1014\n0.23\n1\n15\n0\n186\n0\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\nsurvived\n418\n0.68\nFALSE\n2\nno: 549, yes: 342\n\n\npclass\n0\n1.00\nTRUE\n3\n3: 709, 1: 323, 2: 277\n\n\nsex\n0\n1.00\nFALSE\n2\nmal: 843, fem: 466\n\n\nembarked\n2\n1.00\nFALSE\n3\nS: 914, C: 270, Q: 123\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nage\n263\n0.8\n29.88\n14.41\n0.17\n21.0\n28.00\n39.00\n80.00\n▂▇▅▂▁\n\n\nsib_sp\n0\n1.0\n0.50\n1.04\n0.00\n0.0\n0.00\n1.00\n8.00\n▇▁▁▁▁\n\n\nparch\n0\n1.0\n0.39\n0.87\n0.00\n0.0\n0.00\n0.00\n9.00\n▇▁▁▁▁\n\n\nfare\n1\n1.0\n33.30\n51.76\n0.00\n7.9\n14.45\n31.27\n512.33\n▇▁▁▁▁\n\n\n\n\n\nWe can now create a Task from our data. As we want to classify whether the person survived or not, we will create a TaskClassif. We’ll ignore the ‘titanic_test’ data for now and come back to it later." - }, - { - "objectID": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html#a-first-model", - "href": "gallery/2020-03-12-intro-pipelines-titanic/2020-03-12-mlr3pipelines-titanic-Basics.html#a-first-model", - "title": "A Pipeline for the Titanic Data Set - Basics", - "section": "A first model", - "text": "A first model\nIn order to obtain solutions comparable to official leaderboards, such as the ones available from kaggle, we split the data into train and validation set before doing any further analysis. Here we are using the predefined split used by Kaggle.\n\ntask = as_task_classif(titanic, target = \"survived\", positive = \"yes\")\ntask$set_row_roles(892:1309, \"holdout\")\ntask\n\n (891 x 11)\n* Target: survived\n* Properties: twoclass\n* Features (10):\n - chr (3): cabin, name, ticket\n - dbl (2): age, fare\n - fct (2): embarked, sex\n - int (2): parch, sib_sp\n - ord (1): pclass\n\n\nOur Task currently has \\(3\\) features of type character, which we don’t really know how to handle: “Cabin”, “Name”, “Ticket” and “PassengerId”. Additionally, from our skimr::skim() of the data, we have seen, that they have many unique values (up to 891).\nWe’ll drop them for now and see how we can deal with them later on.\n\ntask$select(cols = setdiff(task$feature_names, c(\"cabin\", \"name\", \"ticket\")))\n\nAdditionally, we create a resampling instance that allows to compare data.\n\ncv3 = rsmp(\"cv\", folds = 3L)$instantiate(task)\n\nTo get a first impression of what performance we can fit a simple decision tree:\n\nlearner = mlr_learners$get(\"classif.rpart\")\n# or shorter:\nlearner = lrn(\"classif.rpart\")\n\nrr = resample(task, learner, cv3, store_models = TRUE)\n\nrr$aggregate(msr(\"classif.acc\"))\n\nclassif.acc \n 0.8013468 \n\n\nSo our model should have a minimal accuracy of 0.80 in order to improve over the simple decision tree. In order to improve more, we might need to do some feature engineering." - }, - { - "objectID": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html", - "href": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html", - "title": "Liver Patient Classification Based on Diagnostic Measures", - "section": "", - "text": "The following examples were created as part of the Introduction to Machine Learning Lecture at LMU Munich. The goal of the project was to create and compare one or several machine learning pipelines for the problem at hand together with exploratory analysis and an exposition of results. The posts were contributed to the mlr3gallery by the authors and edited for better legibility by the editor. We want to thank the authors for allowing us to publish their results. Note, that correctness of the results can not be guaranteed.\n\n\nThis tutorial assumes familiarity with the basics of mlr3tuning and mlr3pipelines. Consult the mlr3book if some aspects are not fully understandable. We load the most important packages for this example.\n\nlibrary(mlr3verse)\nlibrary(dplyr)\nlibrary(tidyr)\nlibrary(DataExplorer)\nlibrary(ggplot2)\nlibrary(gridExtra)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nNote, that expensive calculations are pre-saved in rds files in this tutorial to save computational time.\nMachine learning (ML), a branch of both computer science and statistics, in conjunction with new computing technologies has been transforming research and industries across the board over the past decade. A prime example for this is the healthcare industry, where applications of ML, as well as artificial intelligence in general, have become more and more popular in recent years. One very frequently researched and applied use of ML in the medical field is the area of disease identification and diagnosis. ML technologies have shown potential in detecting anomalies and diseases through pattern recognition, even though an entirely digital diagnosis by a computer is probably still something for the far future. However, suitable and reliable models estimating the risk of diseases could help real doctors make quicker and better decisions today already. In this use case we examined machine learning algorithms and learners for the specific application of liver disease detection. The task is therefore a binary classification task to predict whether a patient has liver disease or not based on some common diagnostic measurements. This report is organized as follows. Section 1 introduces the data and section 2 provides more in-depth data exploration. Section 3 presents learners and their hyperparameter tuning while section 4, dealing with model fitting and benchmarking, presents results and conclusions." - }, - { - "objectID": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#univariate-distribution", - "href": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#univariate-distribution", - "title": "Liver Patient Classification Based on Diagnostic Measures", - "section": "Univariate distribution", - "text": "Univariate distribution\nNext, we looked into the univariate distribution of each of the variables. We began with the target and the only discrete feature, gender, which are both binary.\n\n\n\n\n\n\n\n\n\nThe distribution of the target variable is quite imbalanced, as the barplot shows: the number of patients with and without liver disease equals 416 and 167, respectively. The underrepresentation of a class, in our case those without liver disease, might worsen the performance of ML models. In order to examine this, we additionally fitted the models on a dataset where we randomly over-sampled the minority class, resulting in a perfectly balanced dataset. Furthermore, we applied stratified sampling to ensure the proportion of the classes is maintained during cross-validation.\nThe only discrete feature gender is quite imbalanced, too. As one can see in the next section, this proportion is also observed within each target class. Prior to that, we looked into the distributions of the metric features.\n\n\n\n\n\n\n\n\n\nStrikingly, some of the metric features are extremely right-skewed and contain several extreme values. To reduce the impact of outliers and since some models assume normality of features, we log-transformed these variables." - }, - { - "objectID": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#features-by-class", - "href": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#features-by-class", - "title": "Liver Patient Classification Based on Diagnostic Measures", - "section": "Features by class", - "text": "Features by class\nTo picture the relationship between the target and the features, we analysed the distributions of the features by class. First, we examined the discrete feature gender.\n\n\n\n\n\n\n\n\n\nThe percentage of males in the “disease” class is slightly higher, but overall the difference is small. Besides that, the gender imbalance can be observed in both classes, as we mentioned before. To see the differences in metric features, we compare the following boxplots, where right-skewed features are not log-transformed yet.\n\n\n\n\n\n\n\n\n\nExcept for the total amount of protein, for each feature we obtain differences between the median values of the two classes. Notably, in the case of strongly right-skewed features the “disease” class contains far more extreme values than the “no disease” class, which is probably because of its larger size. This effect is weakened by log-transforming such features, as can be seen in the boxplots below. Moreover, the dispersion in the class “disease” is greater for these features, as the length of the boxes indicates. Overall, the features seem to be correlated to the target, so it makes sense to use them for this task and model their relationship with the target.\n\n\n\n\n\n\n\n\n\nNote, that the same result can be achieved more easily by using PipeOpMutate from mlr3pipelines. This PipeOp provides a smooth implementation to scale numeric features for mlr3 tasks." - }, - { - "objectID": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#correlation", - "href": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#correlation", - "title": "Liver Patient Classification Based on Diagnostic Measures", - "section": "Correlation", - "text": "Correlation\nAs we mentioned in the description of the data, there are features that are indirectly measured by another one. This suggests that they are highly correlated. Some of the models we want to compare assume independent features or have problems with multicollinearity. Therefore, we checked for correlations between features.\n\n\nRegistered S3 method overwritten by 'GGally':\n method from \n +.gg ggplot2\n\n\nWarning in cor(data, use = method[1], method = method[2]): the standard deviation is zero\n\n\n\n\n\n\n\n\n\nFor four of the pairs we obtained a very high correlation coefficient. Looking at these features, it is clear they affect each other. As the complexity of the model should be minimized and due to multicollinearity concerns, we decided to take only one of each pair. When deciding on which features to keep, we chose those that are more specific and relevant regarding liver disease. Therefore, we chose albumin over the ratio between albumin and globulin and also over the total amount of protein. The same argument applies to using the amount of direct bilirubin instead of the total amount of bilirubin. Regarding aspartate transaminase and alanine transaminase, it was not clear which one to use, especially since we have no given real world implementation for the task and no medical training. Since we did not notice any fundamental differences in the data for these two features, we arbitrarily chose aspartate transaminase." - }, - { - "objectID": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#final-dataset", - "href": "gallery/2020-09-11-liver-patient-classification/2020-09-11-liver-patient-classification.html#final-dataset", - "title": "Liver Patient Classification Based on Diagnostic Measures", - "section": "Final Dataset", - "text": "Final Dataset\n\n## Reducing, transforming and scaling dataset\nilpd = ilpd %>%\n select(-total_bilirubin, -alanine_transaminase, -total_protein,\n -albumin_globulin_ratio) %>%\n mutate(\n # Recode gender\n gender = as.numeric(ifelse(gender == \"Female\", 1, 0)),\n # Remove labels for class\n diseased = factor(ifelse(diseased == \"yes\", 1, 0)),\n # Log for features with skewed distributions\n alkaline_phosphatase = log(alkaline_phosphatase),\n aspartate_transaminase = log(aspartate_transaminase),\n direct_bilirubin = log(direct_bilirubin)\n )\n\npo_scale = po(\"scale\")\npo_scale$param_set$values$affect_columns =\n selector_name(c(\"age\", \"direct_bilirubin\", \"alkaline_phosphatase\",\n \"aspartate_transaminase\", \"albumin\"))\n\nLastly, we standardized all metric features, as different ranges and units might weigh features. This is especially important for the k-NN model. The following table shows the final dataset and the transformations we applied. Note: Different from log or other transformation, scaling depends on the data themselves. Scaling data before data are split leads to data leakage, were information of train and test set are shared. As Data Leakage causes higher performance, scaling should always be applied in each data split induced by the ML workflow separately. Therefore we strongly recommend the usage of PipeOpScale in such cases.\n\n\n\nVariable\nTransformation\n\n\n\n\nage\nscaled\n\n\nalbumin\nscaled\n\n\nalkaline_phosphatase\nscaled and log-transformed\n\n\naspartate_transaminase\nscaled and log-transformed\n\n\ndirect_bilirubin\nscaled and log-transformed\n\n\ndiseased\nnone\n\n\ngender\nnone" - }, - { - "objectID": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html", - "href": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html", - "title": "Tuning a Stacked Learner", - "section": "", - "text": "Multilevel stacking is an ensemble technique, where predictions of several learners are added as new features to extend the orginal data on different levels. On each level, the extended data is used to train a new level of learners. This can be repeated for several iterations until a final learner is trained. To avoid overfitting, it is advisable to use test set (out-of-bag) predictions in each level.\nIn this post, a multilevel stacking example will be created using mlr3pipelines and tuned using mlr3tuning . A similar example is available in the mlr3book. However, we additionally explain how to tune the hyperparameters of the whole ensemble and each underlying learner jointly.\nIn our stacking example, we proceed as follows:\n\nLevel 0: Based on the input data, we train three learners (rpart, glmnet and lda) on a sparser feature space obtained using different feature filter methods from mlr3filters to obtain slightly decorrelated predictions. The test set predictions of these learners are attached to the original data (used in level 0) and will serve as input for the learners in level 1.\nLevel 1: We transform this extended data using PCA, on which we then train additional three learners (rpart, glmnet and lda). The test set predictions of the level 1 learners are attached to input data used in level 1.\nFinally, we train a final ranger learner to the data extended by level 1. Note that the number of features selected by the feature filter method in level 0 and the number of principal components retained in level 1 will be jointly tuned with some other hyperparameters of the learners in each level." - }, - { - "objectID": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#prerequisites", - "href": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#prerequisites", - "title": "Tuning a Stacked Learner", - "section": "Prerequisites", - "text": "Prerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners.\n\nlibrary(mlr3verse)\nlibrary(mlr3learners)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nFor the stacking example, we use the sonar classification task:\n\ntask_sonar = tsk(\"sonar\")\ntask_sonar$col_roles$stratum = task_sonar$target_names #stratification" - }, - { - "objectID": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#pipeline-creation", - "href": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#pipeline-creation", - "title": "Tuning a Stacked Learner", - "section": "Pipeline creation", - "text": "Pipeline creation\n\nLevel 0\nAs mentioned, the level 0 learners are rpart, glmnet and lda:\n\nlearner_rpart = lrn(\"classif.rpart\", predict_type = \"prob\")\nlearner_glmnet = lrn(\"classif.glmnet\", predict_type = \"prob\")\nlearner_lda = lrn(\"classif.lda\", predict_type = \"prob\")\n\nTo create the learner out-of-bag predictions, we use PipeOpLearnerCV:\n\ncv1_rpart = po(\"learner_cv\", learner_rpart, id = \"rprt_1\")\ncv1_glmnet = po(\"learner_cv\", learner_glmnet, id = \"glmnet_1\")\ncv1_lda = po(\"learner_cv\", learner_lda, id = \"lda_1\")\n\nA sparser representation of the input data in level 0 is obtained using the following filters:\n\nanova = po(\"filter\", flt(\"anova\"), id = \"filt1\")\nmrmr = po(\"filter\", flt(\"mrmr\"), id = \"filt2\")\nfind_cor = po(\"filter\", flt(\"find_correlation\"), id = \"filt3\")\n\nTo summarize these steps into level 0, we use the gunion() function. The out-of-bag predictions of all level 0 learners is attached using PipeOpFeatureUnion along with the original data passed via PipeOpNOP:\n\nlevel0 = gunion(list(\n anova %>>% cv1_rpart,\n mrmr %>>% cv1_glmnet,\n find_cor %>>% cv1_lda,\n po(\"nop\", id = \"nop1\"))) %>>%\n po(\"featureunion\", id = \"union1\")\n\nWe can have a look at the graph from level 0:\n\nlevel0$plot(html = TRUE)\n\n\n\n\n\n\n\nLevel 1\nNow, we create the level 1 learners:\n\ncv2_rpart = po(\"learner_cv\", learner_rpart , id = \"rprt_2\")\ncv2_glmnet = po(\"learner_cv\", learner_glmnet, id = \"glmnet_2\")\ncv2_lda = po(\"learner_cv\", learner_lda, id = \"lda_2\")\n\nAll level 1 learners will use PipeOpPCA transformed data as input:\n\nlevel1 = level0 %>>%\n po(\"copy\", 4) %>>%\n gunion(list(\n po(\"pca\", id = \"pca2_1\", param_vals = list(scale. = TRUE)) %>>% cv2_rpart,\n po(\"pca\", id = \"pca2_2\", param_vals = list(scale. = TRUE)) %>>% cv2_glmnet,\n po(\"pca\", id = \"pca2_3\", param_vals = list(scale. = TRUE)) %>>% cv2_lda,\n po(\"nop\", id = \"nop2\"))\n ) %>>%\n po(\"featureunion\", id = \"union2\")\n\nWe can have a look at the graph from level 1:\n\nlevel1$plot(html = TRUE)\n\n\n\n\n\nThe out-of-bag predictions of the level 1 learners are attached to the input data from level 1 and a final ranger learner will be trained:\n\nranger_lrn = lrn(\"classif.ranger\", predict_type = \"prob\")\n\nensemble = level1 %>>% ranger_lrn\nensemble$plot(html = TRUE)\n\n\n\n\n\n\n\nDefining the tuning space\nIn order to tune the ensemble’s hyperparameter jointly, we define the search space using ParamSet from the paradox package:\n\nsearch_space_ensemble = ps(\n filt1.filter.nfeat = p_int(5, 50),\n filt2.filter.nfeat = p_int(5, 50),\n filt3.filter.nfeat = p_int(5, 50),\n pca2_1.rank. = p_int(3, 50),\n pca2_2.rank. = p_int(3, 50),\n pca2_3.rank. = p_int(3, 20),\n rprt_1.cp = p_dbl(0.001, 0.1),\n rprt_1.minbucket = p_int(1, 10),\n glmnet_1.alpha = p_dbl(0, 1),\n rprt_2.cp = p_dbl(0.001, 0.1),\n rprt_2.minbucket = p_int(1, 10),\n glmnet_2.alpha = p_dbl(0, 1),\n classif.ranger.mtry = p_int(1, 10),\n classif.ranger.sample.fraction = p_dbl(0.5, 1),\n classif.ranger.num.trees = p_int(50, 200))\n\n\n\nPerformance comparison\nEven with a simple ensemble, there is quite a few things to setup. We compare the performance of the ensemble with a simple tuned ranger learner.\nTo proceed, we convert the ensemble pipeline as a GraphLearner:\n\nlearner_ensemble = as_learner(ensemble)\nlearner_ensemble$id = \"ensemble\"\nlearner_ensemble$predict_type = \"prob\"\n\nWe define the search space for the simple ranger learner:\n\nsearch_space_ranger = ps(\n mtry = p_int(1, 10),\n sample.fraction = p_dbl(0.5, 1),\n num.trees = p_int(50, 200))\n\nFor performance comparison, we use the benchmark() function that requires a design incorporating a list of learners and a list of tasks. Here, we have two learners (the simple ranger learner and the ensemble) and one task. Since we want to tune the simple ranger learner as well as the whole ensemble learner, we need to create an AutoTuner for each learner to be compared. To do so, we need to define a resampling strategy for the tuning in the inner loop (we use 3-fold cross-validation) and for the final evaluation (outer loop) use use holdout validation:\n\ninner_resampling = rsmp(\"cv\", folds = 3)\n\n# AutoTuner for the ensemble learner\nat_1 = auto_tuner(\n method = \"random_search\",\n learner = learner_ensemble,\n resampling = inner_resampling,\n measure = msr(\"classif.auc\"),\n search_space = search_space_ensemble,\n term_evals = 3) # to limit running time\n\n# AutoTuner for the simple ranger learner\nat_2 = auto_tuner(\n method = \"random_search\",\n learner = ranger_lrn,\n resampling = inner_resampling,\n measure = msr(\"classif.auc\"),\n search_space = search_space_ranger,\n term_evals = 3) # to limit running time\n\n# Define the list of learners\nlearners = list(at_1, at_2)\n\n# For benchmarking, we use a simple holdout\nouter_resampling = rsmp(\"holdout\")\nouter_resampling$instantiate(task_sonar)\n\ndesign = benchmark_grid(\n tasks = task_sonar,\n learners = learners,\n resamplings = outer_resampling\n)\n\n\nbmr = benchmark(design, store_models = TRUE)\n\n\n\n\n\n\n\n\nbmr$aggregate(msr(\"classif.auc\"))\n\n\n\n\n\n \n \n nr \n task_id \n learner_id \n resampling_id \n iters \n classif.auc \n \n \n\n \n 1 \n sonar \n ensemble.tuned \n holdout \n 1 \n 0.9020270 \n \n \n 2 \n sonar \n classif.ranger.tuned \n holdout \n 1 \n 0.9113176 \n \n\n\n\n\nFor a more reliable comparison, the number of evaluation of the random search should be increased." - }, - { - "objectID": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#conclusion", - "href": "gallery/2020-04-27-tuning-stacking/2020-04-27-tuning-stacking.html#conclusion", - "title": "Tuning a Stacked Learner", - "section": "Conclusion", - "text": "Conclusion\nThis example shows the versatility of mlr3pipelines. By using more learners, varied representations of the data set as well as more levels, a powerful yet compute hungry pipeline can be created. It is important to note that care should be taken to avoid name clashes of pipeline objects." - }, - { - "objectID": "gallery/2020-03-30-stratification-blocking/2020-03-30-stratification-blocking.html", - "href": "gallery/2020-03-30-stratification-blocking/2020-03-30-stratification-blocking.html", - "title": "Resampling - Stratified, Blocked and Predefined", - "section": "", - "text": "Intro\nWhen evaluating machine learning algorithms through resampling, it is preferable that each train/test partition will be a representative subset of the whole data set. This post covers three ways to achieve such reliable resampling procedures:\n\nStratified resampling for classification problems where each train/test split maintains the target class distribution of the original data set.\nBlock resampling where a grouping factor determines which observations should be together in train/test splits.\nCustom resampling using predefined and manually created folds for the train/test splits.\n\n\n\nPrerequisites\nWe load the most important packages for this post.\n\nlibrary(mlr3verse)\nlibrary(mlbench)\nlibrary(data.table)\n\nWe initialize the random number generator with a fixed seed for reproducibility.\n\nset.seed(7832)\n\n\n\nStratified resampling\nIn classification tasks, the ratio of the target class distribution should be similar in each train/test split, which is achieved by stratification. This is particularly useful in the case of imbalanced classes and small data sets.\nStratification can also be performed with respect to explanatory categorical variables to ensure that all subgroups are represented in all training and test sets.\nIn mlr3, each Task has a slot $col_roles. This slot shows general roles certain features will have throughout different stages of the machine learning process. At least, the $col_roles slot shows which variables will be used as features and as the target. However, the $col_roles slot can be more diverse and some variables might even serve multiple roles. We can specify the variable used for stratification in task$col_roles$stratum. This will be illustrated in the following example using the german_credit data:\n\ntask_gc = tsk(\"german_credit\")\ntask_gc$col_roles\n\n$feature\n [1] \"age\" \"amount\" \"credit_history\" \"duration\" \n [5] \"employment_duration\" \"foreign_worker\" \"housing\" \"installment_rate\" \n [9] \"job\" \"number_credits\" \"other_debtors\" \"other_installment_plans\"\n[13] \"people_liable\" \"personal_status_sex\" \"present_residence\" \"property\" \n[17] \"purpose\" \"savings\" \"status\" \"telephone\" \n\n$target\n[1] \"credit_risk\"\n\n$name\ncharacter(0)\n\n$order\ncharacter(0)\n\n$stratum\ncharacter(0)\n\n$group\ncharacter(0)\n\n$weight\ncharacter(0)\n\n\nWe use the target feature called credit_risk to specify stratification with respect to the target variable:\n\ntask_gc$col_roles$stratum = \"credit_risk\"\n# alternatively task_gc$col_roles$stratum = task_gc$col_roles$target\n\nAfter the specification of task$col_roles$stratum, the active binding task$strata will show the number of observations in each group and the corresponding row id’s:\n\ntask_gc$strata\n\n N row_id\n1: 700 1,3,4,6,7,8,...\n2: 300 2, 5,10,11,12,14,...\n\n\nSpecify 3-fold cross validation and instantiate the resampling on the task:\n\ncv3 = rsmp(\"cv\", folds = 3)\ncv3$instantiate(task_gc)\ncv3$instance\n\n row_id fold\n 1: 7 1\n 2: 8 1\n 3: 9 1\n 4: 17 1\n 5: 22 1\n --- \n 996: 959 3\n 997: 967 3\n 998: 980 3\n 999: 984 3\n1000: 999 3\n\n\nCheck if the target class distribution is similar in each fold:\n\ndt = merge(cv3$instance, task_gc$data()[, row_id := .I], by = \"row_id\")\ndt[, .(class_ratio = sum(credit_risk == \"bad\") /\n sum(credit_risk == \"good\")), by = fold]\n\n fold class_ratio\n1: 2 0.4291845\n2: 3 0.4291845\n3: 1 0.4273504\n\n\nAnd compare it with the target class distribution from the whole data set:\n\ndt[, .(class_ratio = sum(credit_risk == \"bad\") / sum(credit_risk == \"good\"))]\n\n class_ratio\n1: 0.4285714\n\n\nNote that the variable used for stratification does not necessarily have to be the target class. In fact, multiple categorical features can be used for stratification to maintain their frequency distribution in each fold:\n\ntask_gc$col_roles$stratum = c(\"housing\", \"telephone\")\ntask_gc$strata\n\n N row_id\n1: 280 1,13,20,21,26,30,...\n2: 433 2, 3, 7, 9,10,14,...\n3: 47 4, 5, 45, 76,134,192,...\n4: 61 6,19,37,55,63,69,...\n5: 63 8, 48, 60, 72, 96,100,...\n6: 116 11,12,15,22,23,28,...\n\n\nTo illustrate if stratification based on multiple categorical features works, we need to instantiate the CV folds again as we changed the features used for stratification:\n\ncv3$instantiate(task_gc)\ncv3$instance\n\n row_id fold\n 1: 13 1\n 2: 21 1\n 3: 31 1\n 4: 33 1\n 5: 43 1\n --- \n 996: 945 3\n 997: 973 3\n 998: 974 3\n 999: 986 3\n1000: 993 3\n\n\nAgain, we check the relative frequency of observations in each group (combination of housing and telephone) across all folds:\n\ndt = merge(cv3$instance, task_gc$data()[, row_id := .I], by = \"row_id\")\ndt = dt[, .(freq = .N), by = list(fold, housing, telephone)]\ndt = dcast(dt, housing + telephone ~ fold)\n\nUsing 'freq' as value column. Use 'value.var' to override\n\ndt[, c(3:5) := lapply(.SD, function(x) x / sum(x)), .SDcols = 3:5]\ndt\n\n housing telephone 1 2 3\n1: for free no 0.11607143 0.11711712 0.11480363\n2: for free yes (under customer name) 0.06250000 0.06306306 0.06344411\n3: rent no 0.43154762 0.43243243 0.43504532\n4: rent yes (under customer name) 0.27976190 0.27927928 0.28096677\n5: own no 0.04761905 0.04804805 0.04531722\n6: own yes (under customer name) 0.06250000 0.06006006 0.06042296\n\n\nAnd compare it with the relative frequency from the whole data set:\n\ntask_gc$data()[, .(freq = .N / max(.I)),\n by = list(housing, telephone)\n][order(housing, telephone), ]\n\n housing telephone freq\n1: for free no 0.11681772\n2: for free yes (under customer name) 0.06415479\n3: rent no 0.43300000\n4: rent yes (under customer name) 0.28084253\n5: own no 0.04895833\n6: own yes (under customer name) 0.06106106\n\n\nIt is evident that in each fold, the combination of housing and telephone have similar frequencies that also coincide with the frequencies from the whole data set.\n\n\nBlock resampling\nAn additional concern when specifying resampling is respecting the natural grouping of the data. Blocking refers to the situation where subsets of observations belong together and must not be separated during resampling. Hence, for one train/test set pair the entire block is either in the training set or in the test set.\nThe following example is based on the BreastCancer data set from the mlbench package:\n\ndata(BreastCancer, package = \"mlbench\")\ntask_bc = as_task_classif(BreastCancer, target = \"Class\", positive = \"malignant\")\n\nIn the BreastCancer data set, for example, several observations have the same “Id” (Sample code number) which implies these are samples taken from the same patient at different times.\n\n# Let's count how many observation actually have the same Id more than once\nsum(table(BreastCancer$Id) > 1)\n\n[1] 46\n\n\nThere are 46 Id’s with more than one observation (row).\nThe model trained on this data set will be used to predict cancer status of new patients. Hence, we have to make sure that each Id occurs exactly in one fold, so that all observations with the same Id should be either used for training or for evaluating the model. This way, we get less biased performance estimates via k-fold cross validation. The following example will illustrate block cross validation which can be achieved by specifying a blocking factor in the task$col_roles$group slot:\n\n# Use Id column as block factor\ntask_bc$col_roles$group = \"Id\"\n# Remove Id from feature\n# task_bc$col_roles$feature = setdiff(task_bc$col_roles$feature, \"Id\")\ncv5 = rsmp(\"cv\", folds = 5)\nset.seed(123)\ncv5$instantiate(task_bc)\ncv5$instance\n\n row_id fold\n 1: 1016277 1\n 2: 1044572 1\n 3: 1049815 1\n 4: 1050718 1\n 5: 1054590 1\n --- \n641: 1369821 5\n642: 1371026 5\n643: 1371920 5\n644: 714039 5\n645: 841769 5\n\n\nIn this case, the row_id column of the cv5$instance slot refers to values of the grouping variable “Id”. Additionally, the number of rows of the cv5$instance is the same as the number of unique groups:\n\nall(cv5$instance$row_id %in% BreastCancer$Id)\n\n[1] TRUE\n\nnrow(cv5$instance) == length(unique(BreastCancer$Id))\n\n[1] TRUE\n\n\nIf the specified blocking groups are respected, each Id appears only in exactly one fold. To inspect if blocking was successful when generating the folds we count how often each Id appears in a specific fold and print the Ids that appear in more than one fold:\n\ndt = merge(task_bc$data(), cv5$instance, by.x = \"Id\", by.y = \"row_id\")\ndt = dt[, .(unique_folds = length(unique(fold))), by = Id]\ndt[unique_folds > 1, ]\n\nEmpty data.table (0 rows and 2 cols): Id,unique_folds\n\n\nAs expected, the table is empty as there are no Id’s present in more than one fold.\n\n\nResampling with predefined folds\nIn some use cases, it might be necessary to use predefined folds. When using k-fold cross validation without repetition this can be achieved by manually creating a feature used to denote folds and assigning it to the task$col_roles$group slot. First, we create a vector that contains 5 predefined folds:\n\nfolds = sample(rep(1:5, length.out = nrow(BreastCancer)),\n size = nrow(BreastCancer),\n replace = F\n)\nhead(folds, 20)\n\n [1] 2 2 4 1 5 2 5 3 1 5 4 3 3 4 5 3 3 5 2 4\n\ntable(folds)\n\nfolds\n 1 2 3 4 5 \n140 140 140 140 139 \n\n\nThis vector is now added to the data set and will be used as grouping factor just as when defining block resampling:\n\ntask_bc = TaskClassif$new(\n id = \"BreastCancer\",\n backend = data.frame(BreastCancer, foldIds = as.factor(folds)),\n target = \"Class\",\n positive = \"malignant\"\n)\ntask_bc$col_roles$group = \"foldIds\"\n# Remove \"foldIds\" from features\n# task_bc$col_roles$feature = setdiff(task_bc$col_roles$feature, \"foldIds\")\n\nWe now instantiate a 5-fold CV that will respect the predefined folds:\n\ncv5 = rsmp(\"cv\", folds = 5)\ncv5$instantiate(task_bc)\ncv5$instance\n\n row_id fold\n1: 1 1\n2: 2 2\n3: 3 3\n4: 4 4\n5: 5 5\n\n\nSince we have only five predefined folds, the cv5$instance data table has five rows and shows which of our foldIds values (contained in the row_id column) will belong to which instantiated fold. To check if the predefined groups are respected, we count how often each foldIds appears in a specific fold:\n\ndt = merge(task_bc$data(), cv5$instance, by.x = \"foldIds\", by.y = \"row_id\")\ndt[, .(unique_folds = length(unique(fold))), by = foldIds]\n\n foldIds unique_folds\n1: 1 1\n2: 2 1\n3: 3 1\n4: 4 1\n5: 5 1\n\n\nThere are five groups and each foldIds appears only in exactly one fold. This means that each instantiated fold corresponds to one of the predefined folds.\nThe previous example does not cover how to perform repeated k-fold CV or time series CV with predefined indices. This is possible via the mlr_resamplings_custom to which a list of predefined train and test indices can be assigned. In the following example, a custom resampling is created using indices created by caret::createMultiFolds():\n\ntask_gc = tsk(\"german_credit\")\ntrain_ind = caret::createMultiFolds(task_gc$truth(), k = 5, times = 10)\ntest_ind = lapply(train_ind, function(x) setdiff(1:task_gc$nrow, x))\nrc = rsmp(\"custom\")\nrc$instantiate(task_gc, train_ind, test_ind)\n\nWe now check if the instantiated custom resampling contains the intended folds:\n\n# check it for the first fold\nall.equal(train_ind[[1]], rc$train_set(1))\n\n[1] TRUE\n\n# check it for all folds\nunlist(lapply(1:rc$iters, function(i) all.equal(train_ind[[i]], rc$train_set(i))))\n\n [1] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE\n[24] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE\n[47] TRUE TRUE TRUE TRUE\n\n\n\n\nConclusions\nThis post shows how to control the resampling process when using mlr3 in order to account for data specificities." - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html", - "title": "Pipelines, Selectors, Branches", - "section": "", - "text": "mlr3pipelines offers a very flexible way to create data preprocessing steps. This is achieved by a modular approach using PipeOps. For detailed overview check the mlr3book.\nRecommended prior readings:\n\nmlr3pipelines tutorial - german credit\nImpute missing variables .\n\nThis post covers:\n\nHow to apply different preprocessing steps on different features\nHow to branch different preprocessing steps, which allows to select the best performing path\nHow to tune the whole pipeline" - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#prerequisites", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#prerequisites", - "title": "Pipelines, Selectors, Branches", - "section": "Prerequisites", - "text": "Prerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nThe Pima Indian Diabetes classification task will be used.\n\ntask_pima = tsk(\"pima\")\nskimr::skim(task_pima$data())\n\n\nData summary\n\n\nName\ntask_pima$data()\n\n\nNumber of rows\n768\n\n\nNumber of columns\n9\n\n\nKey\nNULL\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n1\n\n\nnumeric\n8\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\ndiabetes\n0\n1\nFALSE\n2\nneg: 500, pos: 268\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nage\n0\n1.00\n33.24\n11.76\n21.00\n24.00\n29.00\n41.00\n81.00\n▇▃▁▁▁\n\n\nglucose\n5\n0.99\n121.69\n30.54\n44.00\n99.00\n117.00\n141.00\n199.00\n▁▇▇▃▂\n\n\ninsulin\n374\n0.51\n155.55\n118.78\n14.00\n76.25\n125.00\n190.00\n846.00\n▇▂▁▁▁\n\n\nmass\n11\n0.99\n32.46\n6.92\n18.20\n27.50\n32.30\n36.60\n67.10\n▅▇▃▁▁\n\n\npedigree\n0\n1.00\n0.47\n0.33\n0.08\n0.24\n0.37\n0.63\n2.42\n▇▃▁▁▁\n\n\npregnant\n0\n1.00\n3.85\n3.37\n0.00\n1.00\n3.00\n6.00\n17.00\n▇▃▂▁▁\n\n\npressure\n35\n0.95\n72.41\n12.38\n24.00\n64.00\n72.00\n80.00\n122.00\n▁▃▇▂▁\n\n\ntriceps\n227\n0.70\n29.15\n10.48\n7.00\n22.00\n29.00\n36.00\n99.00\n▆▇▁▁▁" - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#selection-of-features-for-preprocessing-steps", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#selection-of-features-for-preprocessing-steps", - "title": "Pipelines, Selectors, Branches", - "section": "Selection of features for preprocessing steps", - "text": "Selection of features for preprocessing steps\nSeveral features of the pima task have missing values:\n\ntask_pima$missings()\n\ndiabetes age glucose insulin mass pedigree pregnant pressure triceps \n 0 0 5 374 11 0 0 35 227 \n\n\nA common approach in such situations is to impute the missing values and to add a missing indicator column as explained in the Impute missing variables post. Suppose we want to use\n\nPipeOpImputeHist on features “glucose”, “mass” and “pressure” which have only few missing values and\nPipeOpImputeMedian on features “insulin” and “triceps” which have much more missing values.\n\nIn the following subsections, we show two approaches to implement this.\n\n1. Consider all features and apply the preprocessing step only to certain features\nUsing the affect_columns argument of a PipeOp to define the variables on which a PipeOp will operate with an appropriate Selector function:\n\n# imputes values based on histogram\nimputer_hist = po(\"imputehist\",\n affect_columns = selector_name(c(\"glucose\", \"mass\", \"pressure\")))\n# imputes values using the median\nimputer_median = po(\"imputemedian\",\n affect_columns = selector_name(c(\"insulin\", \"triceps\")))\n# adds an indicator column for each feature with missing values\nmiss_ind = po(\"missind\")\n\nWhen PipeOps are constructed this way, they will perform the specified preprocessing step on the appropriate features and pass all the input features to the subsequent steps:\n\n# no missings in \"glucose\", \"mass\" and \"pressure\"\nimputer_hist$train(list(task_pima))[[1]]$missings()\n\ndiabetes age insulin pedigree pregnant triceps glucose mass pressure \n 0 0 374 0 0 227 0 0 0 \n\n# no missings in \"insulin\" and \"triceps\"\nimputer_median$train(list(task_pima))[[1]]$missings()\n\ndiabetes age glucose mass pedigree pregnant pressure insulin triceps \n 0 0 5 11 0 0 35 0 0 \n\n\nWe construct a pipeline that combines imputer_hist and imputer_median. Here, imputer_hist will impute the features “glucose”, “mass” and “pressure”, and imputer_median will impute “insulin” and “triceps”. In each preprocessing step, all the input features are passed to the next step. In the end, we obtain a data set without missing values:\n\n# combine the two impuation methods\nimpute_graph = imputer_hist %>>% imputer_median\nimpute_graph$plot(html = TRUE)\n\n\n\n\nimpute_graph$train(task_pima)[[1]]$missings()\n\ndiabetes age pedigree pregnant glucose mass pressure insulin triceps \n 0 0 0 0 0 0 0 0 0 \n\n\nThe PipeOpMissInd operator replaces features with missing values with a missing value indicator:\n\nhead(miss_ind$train(list(task_pima))[[1]]$data())\n\n\n\n\n\n \n \n diabetes \n missing_glucose \n missing_insulin \n missing_mass \n missing_pressure \n missing_triceps \n \n \n\n \n pos \n present \n missing \n present \n present \n present \n \n \n neg \n present \n missing \n present \n present \n present \n \n \n pos \n present \n missing \n present \n present \n missing \n \n \n neg \n present \n present \n present \n present \n present \n \n \n pos \n present \n present \n present \n present \n present \n \n \n neg \n present \n missing \n present \n present \n missing \n \n\n\n\n\nObviously, this step can not be applied to the already imputed data as there are no missing values. If we want to combine the previous two imputation steps with a third step that adds missing value indicators, we would need to PipeOpCopy the data two times and supply the first copy to impute_graph and the second copy to miss_ind using gunion(). Finally, the two outputs can be combined with PipeOpFeatureUnion:\n\nimpute_missind = po(\"copy\", 2) %>>%\n gunion(list(impute_graph, miss_ind)) %>>%\n po(\"featureunion\")\nimpute_missind$plot(html = TRUE)\n\n\n\n\n\n\nhead(impute_missind$train(task_pima)[[1]]$data())\n\n\n\n\n\n\n\n\n\n\n2. Select the features for each preprocessing step and apply the preprocessing steps to this subset\nWe can use the PipeOpSelect to select the appropriate features and then apply the desired impute PipeOp on them:\n\nimputer_hist_2 = po(\"select\",\n selector = selector_name(c(\"glucose\", \"mass\", \"pressure\")),\n id = \"slct1\") %>>% # unique id so we can combine it in a pipeline with other select PipeOps\n po(\"imputehist\")\n\nimputer_hist_2$plot(html = TRUE)\n\n\n\n\n\n\nhead(imputer_hist_2$train(task_pima)[[1]]$data())\n\n\n\n\n\n \n \n diabetes \n glucose \n mass \n pressure \n \n \n\n \n pos \n 148 \n 33.6 \n 72 \n \n \n neg \n 85 \n 26.6 \n 66 \n \n \n pos \n 183 \n 23.3 \n 64 \n \n \n neg \n 89 \n 28.1 \n 66 \n \n \n pos \n 137 \n 43.1 \n 40 \n \n \n neg \n 116 \n 25.6 \n 74 \n \n\n\n\n\n\nimputer_median_2 =\n po(\"select\", selector = selector_name(c(\"insulin\", \"triceps\")), id = \"slct2\") %>>%\n po(\"imputemedian\")\n\nhead(imputer_median_2$train(task_pima)[[1]]$data())\n\n\n\n\n\n \n \n diabetes \n insulin \n triceps \n \n \n\n \n pos \n 125 \n 35 \n \n \n neg \n 125 \n 29 \n \n \n pos \n 125 \n 29 \n \n \n neg \n 94 \n 23 \n \n \n pos \n 168 \n 35 \n \n \n neg \n 125 \n 29 \n \n\n\n\n\nTo reproduce the result of the fist example (1.), we need to copy the data four times and apply imputer_hist_2, imputer_median_2 and miss_ind on each of the three copies. The fourth copy is required to select the features without missing values and to append it to the final result. We can do this as follows:\n\nother_features = task_pima$feature_names[task_pima$missings()[-1] == 0]\n\nimputer_missind_2 = po(\"copy\", 4) %>>%\n gunion(list(imputer_hist_2,\n imputer_median_2,\n miss_ind,\n po(\"select\", selector = selector_name(other_features), id = \"slct3\"))) %>>%\n po(\"featureunion\")\n\nimputer_missind_2$plot(html = TRUE)\n\n\n\n\n\n\nhead(imputer_missind_2$train(task_pima)[[1]]$data())\n\n\n\n\n\n\n\n\nNote that when there is one input channel, it is automatically copied as many times as needed for the downstream PipeOps. In other words, the code above works also without po(\"copy\", 4):\n\nimputer_missind_3 = gunion(list(imputer_hist_2,\n imputer_median_2,\n miss_ind,\n po(\"select\", selector = selector_name(other_features), id = \"slct3\"))) %>>%\n po(\"featureunion\")\n\nhead(imputer_missind_3$train(task_pima)[[1]]$data())\n\n\n\n\n\n\n\n\nUsually, po(\"copy\") is required when there are more than one input channels and multiple output channels, and their numbers do not match." - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#branching", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#branching", - "title": "Pipelines, Selectors, Branches", - "section": "Branching", - "text": "Branching\nWe can not know if the combination of a learner with this preprocessing graph will benefit from the imputation steps and the added missing value indicators. Maybe it would have been better to just use imputemedian on all the variables. We could investigate this assumption by adding an alternative path to the graph with the mentioned imputemedian. This is possible using the “branch” PipeOp:\n\nimputer_median_3 = po(\"imputemedian\", id = \"simple_median\") # add the id so it does not clash with `imputer_median`\n\nbranches = c(\"impute_missind\", \"simple_median\") # names of the branches\n\ngraph_branch = po(\"branch\", branches) %>>%\n gunion(list(impute_missind, imputer_median_3)) %>>%\n po(\"unbranch\")\n\ngraph_branch$plot(html = TRUE)" - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#tuning-the-pipeline", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#tuning-the-pipeline", - "title": "Pipelines, Selectors, Branches", - "section": "Tuning the pipeline", - "text": "Tuning the pipeline\nTo finalize the graph, we combine it with a rpart learner:\n\ngraph = graph_branch %>>%\n lrn(\"classif.rpart\")\n\ngraph$plot(html = TRUE)\n\n\n\n\n\nTo define the parameters to be tuned, we first check the available ones in the graph:\n\nas.data.table(graph$param_set)\n\n\n\n\n\n\n\n\nWe decide to jointly tune the \"branch.selection\", \"classif.rpart.cp\" and \"classif.rpart.minbucket\" hyperparameters:\n\nsearch_space = ps(\n branch.selection = p_fct(c(\"impute_missind\", \"simple_median\")),\n classif.rpart.cp = p_dbl(0.001, 0.1),\n classif.rpart.minbucket = p_int(1, 10))\n\nIn order to tune the graph, it needs to be converted to a learner:\n\ngraph_learner = as_learner(graph)\n\ncv3 = rsmp(\"cv\", folds = 3)\n\ncv3$instantiate(task_pima) # to generate folds for cross validation\n\ninstance = tune(\n method = \"random_search\",\n task = task_pima,\n learner = graph_learner,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n search_space = search_space,\n term_evals = 5)\n\nas.data.table(instance$archive)\n\n\n\n\n\n\n\n\nThe best performance in this short tuned experiment was achieved with:\n\ninstance$result\n\n branch.selection classif.rpart.cp classif.rpart.minbucket learner_param_vals x_domain classif.ce\n1: impute_missind 0.02107154 7 0.2460938" - }, - { - "objectID": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#conclusion", - "href": "gallery/2020-04-23-pipelines-selectors-branches/2020-04-23-pipelines-selectors-branches.html#conclusion", - "title": "Pipelines, Selectors, Branches", - "section": "Conclusion", - "text": "Conclusion\nThis post shows ways on how to specify features on which preprocessing steps are to be performed. In addition it shows how to create alternative paths in the learner graph. The preprocessing steps that can be used are not limited to imputation. Check the list of available PipeOp." - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "", - "text": "This use case shows how to use the basic mlr3 package on the iris Task, so it’s our “Hello World” example. It assumes no prior knowledge in ML or mlr3. You can find most of the content here also in the mlr3book in a more detailed way. Hence we will not make a lot of general comments, but keep it hands-on and short.\nThe following operations are shown:\n\nCreating Tasks and Learners\nTraining and predicting\nResampling / cross-validation\nInstalling more Learners\nBenchmarking to compare multiple Learners" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#loading-basic-packages", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#loading-basic-packages", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Loading basic packages", - "text": "Loading basic packages\nWe load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners.\n\nlibrary(mlr3verse)\nlibrary(mlr3learners)\n\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#creating-tasks-and-learners", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#creating-tasks-and-learners", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Creating tasks and learners", - "text": "Creating tasks and learners\nLet’s work on the canonical, simple iris data set, and try out some ML algorithms. We will start by using a decision tree with default settings.\n\n# creates mlr3 task from scratch, from a data.frame\n# 'target' names the column in the dataset we want to learn to predict\ntask = as_task_classif(iris, target = \"Species\")\n# in this case we could also take the iris example from mlr3's dictionary of shipped example tasks\n# 2 equivalent calls to create a task. The second is just sugar for the user.\ntask = mlr_tasks$get(\"iris\")\ntask = tsk(\"iris\")\nprint(task)\n\n (150 x 5): Iris Flowers\n* Target: Species\n* Properties: multiclass\n* Features (4):\n - dbl (4): Petal.Length, Petal.Width, Sepal.Length, Sepal.Width\n\n# create learner from dictionary of mlr3learners\n# 2 equivalent calls:\nlearner_1 = mlr_learners$get(\"classif.rpart\")\nlearner_1 = lrn(\"classif.rpart\")\nprint(learner_1)\n\n: Classification Tree\n* Model: -\n* Parameters: xval=0\n* Packages: mlr3, rpart\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, multiclass, selected_features, twoclass, weights" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#train-and-predict", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#train-and-predict", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Train and predict", - "text": "Train and predict\nNow the usual ML operations: Train on some observations, predict on others.\n\n# train learner on subset of task\nlearner_1$train(task, row_ids = 1:120)\n# this is what the decision tree looks like\nprint(learner_1$model)\n\nn= 120 \n\nnode), split, n, loss, yval, (yprob)\n * denotes terminal node\n\n1) root 120 70 setosa (0.41666667 0.41666667 0.16666667) \n 2) Petal.Length< 2.45 50 0 setosa (1.00000000 0.00000000 0.00000000) *\n 3) Petal.Length>=2.45 70 20 versicolor (0.00000000 0.71428571 0.28571429) \n 6) Petal.Length< 4.95 49 1 versicolor (0.00000000 0.97959184 0.02040816) *\n 7) Petal.Length>=4.95 21 2 virginica (0.00000000 0.09523810 0.90476190) *\n\n# predict using observations from task\nprediction = learner_1$predict(task, row_ids = 121:150)\n# predict using \"new\" observations from an external data.frame\nprediction = learner_1$predict_newdata(newdata = iris[121:150, ])\nprint(prediction)\n\n for 30 observations:\n row_ids truth response\n 1 virginica virginica\n 2 virginica versicolor\n 3 virginica virginica\n--- \n 28 virginica virginica\n 29 virginica virginica\n 30 virginica virginica" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#evaluation", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#evaluation", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Evaluation", - "text": "Evaluation\nLet’s score our Prediction object with some metrics. And take a deeper look by inspecting the confusion matrix.\n\nhead(as.data.table(mlr_measures))\n\n key label task_type packages predict_type task_properties\n1: aic Akaika Information Criterion mlr3 response \n2: bic Bayesian Information Criterion mlr3 response \n3: classif.acc Classification Accuracy classif mlr3,mlr3measures response \n4: classif.auc Area Under the ROC Curve classif mlr3,mlr3measures prob twoclass\n5: classif.bacc Balanced Accuracy classif mlr3,mlr3measures response \n6: classif.bbrier Binary Brier Score classif mlr3,mlr3measures prob twoclass\n\nscores = prediction$score(msr(\"classif.acc\"))\nprint(scores)\n\nclassif.acc \n 0.8333333 \n\nscores = prediction$score(msrs(c(\"classif.acc\", \"classif.ce\")))\nprint(scores)\n\nclassif.acc classif.ce \n 0.8333333 0.1666667 \n\ncm = prediction$confusion\nprint(cm)\n\n truth\nresponse setosa versicolor virginica\n setosa 0 0 0\n versicolor 0 0 5\n virginica 0 0 25" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#changing-hyperpars", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#changing-hyperpars", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Changing hyperpars", - "text": "Changing hyperpars\nThe Learner contains information about all parameters that can be configured, including data type, constraints, defaults, etc. We can change the hyperparameters either during construction of later through an active binding.\n\nas.data.table(learner_1$param_set)\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n cp \n ParamDbl \n 0 \n 1 \n Inf \n \n \n keep_model \n ParamLgl \n NA \n NA \n 2 \n \n \n maxcompete \n ParamInt \n 0 \n Inf \n Inf \n \n \n maxdepth \n ParamInt \n 1 \n 30 \n 30 \n \n \n maxsurrogate \n ParamInt \n 0 \n Inf \n Inf \n \n \n minbucket \n ParamInt \n 1 \n Inf \n Inf \n \n \n minsplit \n ParamInt \n 1 \n Inf \n Inf \n \n \n surrogatestyle \n ParamInt \n 0 \n 1 \n 2 \n \n \n usesurrogate \n ParamInt \n 0 \n 2 \n 3 \n \n \n xval \n ParamInt \n 0 \n Inf \n Inf \n \n\n\n\n\n\nlearner_2 = lrn(\"classif.rpart\", predict_type = \"prob\", minsplit = 50)\nlearner_2$param_set$values$minsplit = 50" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#resampling", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#resampling", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Resampling", - "text": "Resampling\nResampling simply repeats the train-predict-score loop and collects all results in a nice data.table::data.table().\n\ncv10 = rsmp(\"cv\", folds = 10)\nrr = resample(task, learner_1, cv10)\nprint(rr)\n\n of 10 iterations\n* Task: iris\n* Learner: classif.rpart\n* Warnings: 0 in 0 iterations\n* Errors: 0 in 0 iterations\n\n\n\nrr$score(msrs(c(\"classif.acc\", \"classif.ce\")))\n\n\n\n\n\n \n \n iteration \n task_id \n learner_id \n resampling_id \n classif.ce \n \n \n\n \n 1 \n iris \n classif.rpart \n cv \n 0.1333333 \n \n \n 2 \n iris \n classif.rpart \n cv \n 0.0666667 \n \n \n 3 \n iris \n classif.rpart \n cv \n 0.0666667 \n \n \n 4 \n iris \n classif.rpart \n cv \n 0.1333333 \n \n \n 5 \n iris \n classif.rpart \n cv \n 0.0000000 \n \n \n 6 \n iris \n classif.rpart \n cv \n 0.1333333 \n \n \n 7 \n iris \n classif.rpart \n cv \n 0.0666667 \n \n \n 8 \n iris \n classif.rpart \n cv \n 0.0666667 \n \n \n 9 \n iris \n classif.rpart \n cv \n 0.0000000 \n \n \n 10 \n iris \n classif.rpart \n cv \n 0.0000000 \n \n\n\n\n\n\n# get all predictions nicely concatenated in a table\nprediction = rr$prediction()\nhead(as.data.table(prediction))\n\n\n\n\n\n \n \n row_ids \n truth \n response \n \n \n\n \n 25 \n setosa \n setosa \n \n \n 53 \n versicolor \n virginica \n \n \n 54 \n versicolor \n versicolor \n \n \n 59 \n versicolor \n versicolor \n \n \n 60 \n versicolor \n versicolor \n \n \n 65 \n versicolor \n versicolor \n \n\n\n\n\n\ncm = prediction$confusion\nprint(cm)\n\n truth\nresponse setosa versicolor virginica\n setosa 50 0 0\n versicolor 0 45 5\n virginica 0 5 45" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#populating-the-learner-dictionary", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#populating-the-learner-dictionary", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Populating the learner dictionary", - "text": "Populating the learner dictionary\nmlr3learners ships out with a dozen different popular Learners. We can list them from the dictionary. If we want more, we can install an extension package, mlr3extralearners, from GitHub. Note how after the installation the dictionary increases in size.\n\nhead(as.data.table(mlr_learners)[, c(\"key\", \"packages\")])\n\n key packages\n1: classif.AdaBoostM1 mlr3,mlr3extralearners,RWeka\n2: classif.C50 mlr3,mlr3extralearners,C50\n3: classif.IBk mlr3,mlr3extralearners,RWeka\n4: classif.J48 mlr3,mlr3extralearners,RWeka\n5: classif.JRip mlr3,mlr3extralearners,RWeka\n6: classif.LMT mlr3,mlr3extralearners,RWeka\n\n# remotes::install_github(\"mlr-org/mlr3extralearners\")\nlibrary(mlr3extralearners)\nprint(as.data.table(mlr_learners)[, c(\"key\", \"packages\")])\n\n key packages\n 1: classif.AdaBoostM1 mlr3,mlr3extralearners,RWeka\n 2: classif.C50 mlr3,mlr3extralearners,C50\n 3: classif.IBk mlr3,mlr3extralearners,RWeka\n 4: classif.J48 mlr3,mlr3extralearners,RWeka\n 5: classif.JRip mlr3,mlr3extralearners,RWeka\n --- \n131: surv.ranger mlr3,mlr3proba,mlr3extralearners,ranger\n132: surv.rfsrc mlr3,mlr3proba,mlr3extralearners,randomForestSRC,pracma\n133: surv.rpart mlr3,mlr3proba,rpart,distr6,survival\n134: surv.svm mlr3,mlr3proba,mlr3extralearners,survivalsvm\n135: surv.xgboost mlr3,mlr3proba,mlr3extralearners,xgboost" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#benchmarking-multiple-learners", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#benchmarking-multiple-learners", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Benchmarking multiple learners", - "text": "Benchmarking multiple learners\nThe benchmark function can conveniently compare `r ref(“Learner”, “Learners”) on the same dataset(s).\n\nlearners = list(learner_1, learner_2, lrn(\"classif.randomForest\"))\ngrid = benchmark_grid(task, learners, cv10)\nbmr = benchmark(grid)\nprint(bmr)\n\n of 30 rows with 3 resampling runs\n nr task_id learner_id resampling_id iters warnings errors\n 1 iris classif.rpart cv 10 0 0\n 2 iris classif.rpart cv 10 0 0\n 3 iris classif.randomForest cv 10 0 0\n\nprint(bmr$aggregate(measures = msrs(c(\"classif.acc\", \"classif.ce\"))))\n\n nr resample_result task_id learner_id resampling_id iters classif.acc classif.ce\n1: 1 iris classif.rpart cv 10 0.9200000 0.08000000\n2: 2 iris classif.rpart cv 10 0.9333333 0.06666667\n3: 3 iris classif.randomForest cv 10 0.9400000 0.06000000" - }, - { - "objectID": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#conclusion", - "href": "gallery/2020-03-18-iris-mlr3-basics/2020-03-18-iris-mlr3-basics.html#conclusion", - "title": "mlr3 Basics on “Iris” - Hello World!", - "section": "Conclusion", - "text": "Conclusion\nWe left out a lot of details and other features. If you want to know more, read the mlr3book and the documentation of the mentioned packages." - }, - { - "objectID": "gallery/2020-02-01-tuning-multiplexer/2020-02-01-tuning-multiplexer.html", - "href": "gallery/2020-02-01-tuning-multiplexer/2020-02-01-tuning-multiplexer.html", - "title": "Tuning Over Multiple Learners", - "section": "", - "text": "This use case shows how to tune over multiple learners for a single task. You will learn the following:\n\nBuild a pipeline that can switch between multiple learners\nDefine the hyperparameter search space for the pipeline\nRun a random or grid search (or any other tuner, always works the same)\nRun nested resampling for unbiased performance estimates\n\nThis is an advanced use case. What should you know before:\n\nmlr3 basics\nmlr3tuning basics, especially AutoTuner\nmlr3pipelines, especially branching\n\n\nThe Setup\nAssume, you are given some ML task and what to compare a couple of learners, probably because you want to select the best of them at the end of the analysis. That’s a super standard scenario, it actually sounds so common that you might wonder: Why an (advanced) blog post about this? With pipelines? We will consider 2 cases: (a) Running the learners in their default, so without tuning, and (b) with tuning.\nWe load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners.\n\nlibrary(mlr3verse)\nlibrary(mlr3tuning)\nlibrary(mlr3learners)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nLet’s define our learners.\n\nlearners = list(\n lrn(\"classif.xgboost\", id = \"xgb\", eval_metric = \"logloss\"),\n lrn(\"classif.ranger\", id = \"rf\")\n)\nlearners_ids = sapply(learners, function(x) x$id)\n\ntask = tsk(\"sonar\") # some random data for this demo\ninner_cv2 = rsmp(\"cv\", folds = 2) # inner loop for nested CV\nouter_cv5 = rsmp(\"cv\", folds = 5) # outer loop for nested CV\n\n\n\nDefault Parameters\n\n\nThe Benchmark-Table Approach\nAssume we don’t want to perform tuning and or with running all learner in their respective defaults. Simply run benchmark on the learners and the tasks. That tabulates our results nicely and shows us what works best.\n\ngrid = benchmark_grid(task, learners, outer_cv5)\nbmr = benchmark(grid)\nbmr$aggregate(measures = msr(\"classif.ce\"))\n\n nr resample_result task_id learner_id resampling_id iters classif.ce\n1: 1 sonar xgb cv 5 0.2736353\n2: 2 sonar rf cv 5 0.1632985\n\n\n\n\nThe Pipelines Approach\nOk, why would we ever want to change the simple approach above - and use pipelines / tuning for this? Three reasons:\n\nWhat we are doing with benchmark() is actually statistically flawed, insofar if we report the error of the numerically best method from the benchmark table as its estimated future performance. If we do that we have “optimized on the CV” (we basically ran a grid search over our learners!) and we know that this is will produce optimistically biased results. NB: This is a somewhat ridiculous criticism if we are going over only a handful of options, and the bias will be very small. But it will be noticeable if we do this over hundreds of learners, so it is important to understand the underlying problem. This is a somewhat subtle point, and this gallery post is more about technical hints for mlr3, so we will stop this discussion here.\nFor some tuning algorithms, you might have a chance to more efficiently select from the set of algorithms than running the full benchmark. Because of the categorical nature of the problem, you will not be able to learn stuff like “If learner A works bad, I don’t have to try learner B”, but you can potentially save some resampling iterations. Assume you have so select from 100 candidates, experiments are expensive, and you use a 20-fold CV. If learner A has super-bad results in the first 5 folds of the CV, you might already want to stop here. “Racing” would be such a tuning algorithm.\nIt helps us to foreshadow what comes later in this post where we tune the learners.\n\nThe pipeline just has a single purpose in this example: It should allow us to switch between different learners, depending on a hyperparameter. The pipe consists of three elements:\n\nbranch pipes incoming data to one of the following elements, on different data channels. We can name these channel on construction with options.\nour learners (combined with gunion())\nunbranch combines the forked paths at the end.\n\n\ngraph =\n po(\"branch\", options = learners_ids) %>>%\n gunion(lapply(learners, po)) %>>%\n po(\"unbranch\")\ngraph$plot(html = TRUE)\n\n\n\n\n\nThe pipeline has now quite a lot of available hyperparameters. It includes all hyperparameters from all contained learners. But as we don’t tune them here (yet), we don’t care (yet). But the first hyperparameter is special. branch.selection controls over which (named) branching channel our data flows.\n\ngraph$param_set$ids()\n\n [1] \"branch.selection\" \"xgb.alpha\" \"xgb.approxcontrib\" \n [4] \"xgb.base_score\" \"xgb.booster\" \"xgb.callbacks\" \n [7] \"xgb.colsample_bylevel\" \"xgb.colsample_bynode\" \"xgb.colsample_bytree\" \n[10] \"xgb.disable_default_eval_metric\" \"xgb.early_stopping_rounds\" \"xgb.eta\" \n[13] \"xgb.eval_metric\" \"xgb.feature_selector\" \"xgb.feval\" \n[16] \"xgb.gamma\" \"xgb.grow_policy\" \"xgb.interaction_constraints\" \n[19] \"xgb.iterationrange\" \"xgb.lambda\" \"xgb.lambda_bias\" \n[22] \"xgb.max_bin\" \"xgb.max_delta_step\" \"xgb.max_depth\" \n[25] \"xgb.max_leaves\" \"xgb.maximize\" \"xgb.min_child_weight\" \n[28] \"xgb.missing\" \"xgb.monotone_constraints\" \"xgb.normalize_type\" \n[31] \"xgb.nrounds\" \"xgb.nthread\" \"xgb.ntreelimit\" \n[34] \"xgb.num_parallel_tree\" \"xgb.objective\" \"xgb.one_drop\" \n[37] \"xgb.outputmargin\" \"xgb.predcontrib\" \"xgb.predictor\" \n[40] \"xgb.predinteraction\" \"xgb.predleaf\" \"xgb.print_every_n\" \n[43] \"xgb.process_type\" \"xgb.rate_drop\" \"xgb.refresh_leaf\" \n[46] \"xgb.reshape\" \"xgb.seed_per_iteration\" \"xgb.sampling_method\" \n[49] \"xgb.sample_type\" \"xgb.save_name\" \"xgb.save_period\" \n[52] \"xgb.scale_pos_weight\" \"xgb.skip_drop\" \"xgb.strict_shape\" \n[55] \"xgb.subsample\" \"xgb.top_k\" \"xgb.training\" \n[58] \"xgb.tree_method\" \"xgb.tweedie_variance_power\" \"xgb.updater\" \n[61] \"xgb.verbose\" \"xgb.watchlist\" \"xgb.xgb_model\" \n[64] \"rf.alpha\" \"rf.always.split.variables\" \"rf.class.weights\" \n[67] \"rf.holdout\" \"rf.importance\" \"rf.keep.inbag\" \n[70] \"rf.max.depth\" \"rf.min.node.size\" \"rf.min.prop\" \n[73] \"rf.minprop\" \"rf.mtry\" \"rf.mtry.ratio\" \n[76] \"rf.num.random.splits\" \"rf.num.threads\" \"rf.num.trees\" \n[79] \"rf.oob.error\" \"rf.regularization.factor\" \"rf.regularization.usedepth\" \n[82] \"rf.replace\" \"rf.respect.unordered.factors\" \"rf.sample.fraction\" \n[85] \"rf.save.memory\" \"rf.scale.permutation.importance\" \"rf.se.method\" \n[88] \"rf.seed\" \"rf.split.select.weights\" \"rf.splitrule\" \n[91] \"rf.verbose\" \"rf.write.forest\" \n\ngraph$param_set$params$branch.selection\n\n id class lower upper levels default\n1: branch.selection ParamFct NA NA xgb,rf \n\n\nWe can now tune over this pipeline, and probably running grid search seems a good idea to “touch” every available learner. NB: We have now written down in (much more complicated code) what we did before with benchmark.\n\ngraph_learner = as_learner(graph)\ngraph_learner$id = \"g\"\ngraph_learner$param_set$values$branch.selection = to_tune(levels = c(\"rf\", \"xgb\"))\n\ninstance = tune(\n method = \"grid_search\",\n task = task,\n learner = graph_learner,\n resampling = inner_cv2,\n measure = msr(\"classif.ce\"))\n\nas.data.table(instance$archive)\n\n\n\n\n\n \n \n branch.selection \n classif.ce \n x_domain_branch.selection \n runtime_learners \n timestamp \n batch_nr \n warnings \n errors \n \n \n\n \n rf \n 0.1778846 \n rf \n 0.120 \n 2022-09-17 18:34:29 \n 1 \n 0 \n 0 \n \n \n xgb \n 0.3269231 \n xgb \n 0.058 \n 2022-09-17 18:34:30 \n 2 \n 0 \n 0 \n \n\n\n\n\nBut: Via this approach we can now get unbiased performance results via nested resampling and using the AutoTuner (which would make much more sense if we would select from 100 models and not 2).\n\nat = auto_tuner(\n method = \"grid_search\",\n learner = graph_learner,\n resampling = inner_cv2,\n measure = msr(\"classif.ce\"),\n)\n\nrr = resample(task, at, outer_cv5, store_models = TRUE)\n\n# access inner tuning result\nextract_inner_tuning_results(rr)\n\n iteration branch.selection classif.ce learner_param_vals x_domain task_id learner_id resampling_id\n1: 1 rf 0.2228916 sonar g.tuned cv\n2: 2 rf 0.1746988 sonar g.tuned cv\n3: 3 rf 0.3012048 sonar g.tuned cv\n4: 4 rf 0.2634825 sonar g.tuned cv\n5: 5 rf 0.2754590 sonar g.tuned cv\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n iteration \n branch.selection \n classif.ce \n task_id \n learner_id \n resampling_id \n \n \n\n \n 1 \n rf \n 0.2228916 \n sonar \n g.tuned \n cv \n \n \n 2 \n rf \n 0.1746988 \n sonar \n g.tuned \n cv \n \n \n 3 \n rf \n 0.3012048 \n sonar \n g.tuned \n cv \n \n \n 4 \n rf \n 0.2634825 \n sonar \n g.tuned \n cv \n \n \n 5 \n rf \n 0.2754590 \n sonar \n g.tuned \n cv \n \n\n\n\n\n\n# access inner tuning archives\nextract_inner_tuning_archives(rr)\n\n\n\n\n\n\n\n\n\n\nModel-Selection and Tuning with Pipelines\nNow let’s select from our given set of models and tune their hyperparameters. One way to do this is to define a search space for each individual learner, wrap them all with the AutoTuner, then call benchmark() on them. As this is pretty standard, we will skip this here, and show an even neater option, where you can tune over models and hyperparameters in one go. If you have quite a large space of potential learners and combine this with an efficient tuning algorithm, this can save quite some time in tuning as you can learn during optimization which options work best and focus on them. NB: Many AutoML systems work in a very similar way.\n\n\nDefine the Search Space\nRemember, that the pipeline contains a joint set of all contained hyperparameters. Prefixed with the respective PipeOp ID, to make names unique.\n\nas.data.table(graph$param_set)\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n branch.selection \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.alpha \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.approxcontrib \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.base_score \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n xgb.booster \n ParamFct \n NA \n NA \n 3 \n \n \n xgb.callbacks \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.colsample_bylevel \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.colsample_bynode \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.colsample_bytree \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.disable_default_eval_metric \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.early_stopping_rounds \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.eta \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.eval_metric \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.feature_selector \n ParamFct \n NA \n NA \n 5 \n \n \n xgb.feval \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.gamma \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.grow_policy \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.interaction_constraints \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.iterationrange \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.lambda \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.lambda_bias \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.max_bin \n ParamInt \n 2 \n Inf \n Inf \n \n \n xgb.max_delta_step \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.max_depth \n ParamInt \n 0 \n Inf \n Inf \n \n \n xgb.max_leaves \n ParamInt \n 0 \n Inf \n Inf \n \n \n xgb.maximize \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.min_child_weight \n ParamDbl \n 0 \n Inf \n Inf \n \n \n xgb.missing \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n xgb.monotone_constraints \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.normalize_type \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.nrounds \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.nthread \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.ntreelimit \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.num_parallel_tree \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.objective \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.one_drop \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.outputmargin \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.predcontrib \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.predictor \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.predinteraction \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.predleaf \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.print_every_n \n ParamInt \n 1 \n Inf \n Inf \n \n \n xgb.process_type \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.rate_drop \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.refresh_leaf \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.reshape \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.seed_per_iteration \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.sampling_method \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.sample_type \n ParamFct \n NA \n NA \n 2 \n \n \n xgb.save_name \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.save_period \n ParamInt \n 0 \n Inf \n Inf \n \n \n xgb.scale_pos_weight \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n xgb.skip_drop \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.strict_shape \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.subsample \n ParamDbl \n 0 \n 1 \n Inf \n \n \n xgb.top_k \n ParamInt \n 0 \n Inf \n Inf \n \n \n xgb.training \n ParamLgl \n NA \n NA \n 2 \n \n \n xgb.tree_method \n ParamFct \n NA \n NA \n 5 \n \n \n xgb.tweedie_variance_power \n ParamDbl \n 1 \n 2 \n Inf \n \n \n xgb.updater \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.verbose \n ParamInt \n 0 \n 2 \n 3 \n \n \n xgb.watchlist \n ParamUty \n NA \n NA \n Inf \n \n \n xgb.xgb_model \n ParamUty \n NA \n NA \n Inf \n \n \n rf.alpha \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n rf.always.split.variables \n ParamUty \n NA \n NA \n Inf \n \n \n rf.class.weights \n ParamUty \n NA \n NA \n Inf \n \n \n rf.holdout \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.importance \n ParamFct \n NA \n NA \n 4 \n \n \n rf.keep.inbag \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.max.depth \n ParamInt \n 0 \n Inf \n Inf \n \n \n rf.min.node.size \n ParamInt \n 1 \n Inf \n Inf \n \n \n rf.min.prop \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n rf.minprop \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n rf.mtry \n ParamInt \n 1 \n Inf \n Inf \n \n \n rf.mtry.ratio \n ParamDbl \n 0 \n 1 \n Inf \n \n \n rf.num.random.splits \n ParamInt \n 1 \n Inf \n Inf \n \n \n rf.num.threads \n ParamInt \n 1 \n Inf \n Inf \n \n \n rf.num.trees \n ParamInt \n 1 \n Inf \n Inf \n \n \n rf.oob.error \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.regularization.factor \n ParamUty \n NA \n NA \n Inf \n \n \n rf.regularization.usedepth \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.replace \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.respect.unordered.factors \n ParamFct \n NA \n NA \n 3 \n \n \n rf.sample.fraction \n ParamDbl \n 0 \n 1 \n Inf \n \n \n rf.save.memory \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.scale.permutation.importance \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.se.method \n ParamFct \n NA \n NA \n 2 \n \n \n rf.seed \n ParamInt \n -Inf \n Inf \n Inf \n \n \n rf.split.select.weights \n ParamUty \n NA \n NA \n Inf \n \n \n rf.splitrule \n ParamFct \n NA \n NA \n 3 \n \n \n rf.verbose \n ParamLgl \n NA \n NA \n 2 \n \n \n rf.write.forest \n ParamLgl \n NA \n NA \n 2 \n \n\n\n\n\nWe decide to tune the mtry parameter of the random forest and the nrounds parameter of xgboost. Additionally, we tune branching parameter that selects our learner.\nWe also have to reflect the hierarchical order of the parameter sets (admittedly, this is somewhat inconvenient). We can only set the mtry value if the pipe is configured to use the random forest (ranger). The same applies for the xgboost parameter.\n\nsearch_space = ps(\n branch.selection = p_fct(c(\"rf\", \"xgb\")),\n rf.mtry = p_int(1L, 20L, depends = branch.selection == \"rf\"),\n xgb.nrounds = p_int(1, 500, depends = branch.selection == \"xgb\"))\n\n\n\nTune the Pipeline with a Random Search\nVery similar code as before, we just swap out the search space. And now use random search.\n\ngraph_learner = as_learner(graph)\ngraph_learner$id = \"g\"\n\ninstance = tune(\n method = \"random_search\",\n task = task,\n learner = graph_learner,\n resampling = inner_cv2,\n measure = msr(\"classif.ce\"),\n term_evals = 10\n)\n\nas.data.table(instance$archive)\n\n branch.selection classif.ce x_domain_branch.selection runtime_learners timestamp batch_nr warnings errors\n 1: xgb 0.3509615 xgb 0.062 2022-09-17 18:34:34 1 0 0\n 2: xgb 0.3557692 xgb 0.121 2022-09-17 18:34:34 2 0 0\n 3: xgb 0.3509615 xgb 0.237 2022-09-17 18:34:35 3 0 0\n 4: xgb 0.3509615 xgb 0.058 2022-09-17 18:34:35 4 0 0\n 5: rf 0.2451923 rf 0.119 2022-09-17 18:34:35 5 0 0\n 6: xgb 0.3461538 xgb 0.060 2022-09-17 18:34:35 6 0 0\n 7: rf 0.2596154 rf 0.119 2022-09-17 18:34:35 7 0 0\n 8: xgb 0.3557692 xgb 0.060 2022-09-17 18:34:35 8 0 0\n 9: rf 0.2403846 rf 0.118 2022-09-17 18:34:36 9 0 0\n10: xgb 0.3509615 xgb 0.058 2022-09-17 18:34:36 10 0 0\n resample_result\n 1: \n 2: \n 3: \n 4: \n 5: \n 6: \n 7: \n 8: \n 9: \n10: \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nThe following shows a quick way to visualize the tuning results.\n\n# autoplot(instance, cols_x = c(\"xgb.nrounds\",\"rf.mtry\"))\nautoplot(instance)\n\n\n\n\n\n\n\n\nNested resampling, now really needed:\n\nrr = tune_nested(\n method = \"grid_search\",\n task = task,\n learner = graph_learner,\n inner_resampling = inner_cv2,\n outer_resampling = outer_cv5,\n measure = msr(\"classif.ce\"),\n term_evals = 10L)\n\n\n\n\n\n\n\n\n# access inner tuning result\nextract_inner_tuning_results(rr)\n\n\n\n\n\n \n \n iteration \n branch.selection \n classif.ce \n learner_param_vals \n x_domain \n task_id \n learner_id \n resampling_id \n \n \n\n \n 1 \n rf \n 0.1626506 \n logloss, 1 , 1 , 0 , 1 , rf \n rf \n sonar \n g.tuned \n cv \n \n \n 2 \n rf \n 0.1927711 \n logloss, 1 , 1 , 0 , 1 , rf \n rf \n sonar \n g.tuned \n cv \n \n \n 3 \n rf \n 0.2168675 \n logloss, 1 , 1 , 0 , 1 , rf \n rf \n sonar \n g.tuned \n cv \n \n \n 4 \n rf \n 0.2395295 \n logloss, 1 , 1 , 0 , 1 , rf \n rf \n sonar \n g.tuned \n cv \n \n \n 5 \n rf \n 0.2155049 \n logloss, 1 , 1 , 0 , 1 , rf \n rf \n sonar \n g.tuned \n cv" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html", - "title": "Feature Selection on the Titanic Data Set", - "section": "", - "text": "In this tutorial, we introduce the mlr3fselect package by comparing feature selection methods on the Titanic disaster data set. The objective of feature selection is to enhance the interpretability of models, speed up the learning process and increase the predictive performance.\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\nlibrary(mlr3fselect)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#titanic-data-set", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#titanic-data-set", - "title": "Feature Selection on the Titanic Data Set", - "section": "Titanic Data Set", - "text": "Titanic Data Set\nThe Titanic data set contains data for 887 Titanic passengers, including whether they survived when the Titanic sank. Our goal will be to predict the survival of the Titanic passengers.\nAfter loading the data set from the mlr3data package, we impute the missing age values with the median age of the passengers, set missing embarked values to \"s\" and remove character features. We could use feature engineering to create new features from the character features, however we want to focus on feature selection in this tutorial.\nIn addition to the survived column, the reduced data set contains the following attributes for each passenger:\n\n\n\nFeature\nDescription\n\n\n\n\nage\nAge\n\n\nsex\nSex\n\n\nsib_sp\nNumber of siblings / spouses aboard\n\n\nparch\nNumber of parents / children aboard\n\n\nfare\nAmount paid for the ticket\n\n\npc_class\nPassenger class\n\n\nembarked\nPort of embarkation\n\n\n\n\nlibrary(mlr3data)\n\ndata(\"titanic\", package = \"mlr3data\")\ntitanic$age[is.na(titanic$age)] = median(titanic$age, na.rm = TRUE)\ntitanic$embarked[is.na(titanic$embarked)] = \"S\"\ntitanic$ticket = NULL\ntitanic$name = NULL\ntitanic$cabin = NULL\ntitanic = titanic[!is.na(titanic$survived),]\n\nWe construct a binary classification task.\n\ntask = as_task_classif(titanic, target = \"survived\", positive = \"yes\")" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#model", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#model", - "title": "Feature Selection on the Titanic Data Set", - "section": "Model", - "text": "Model\nWe use the logistic regression learner provided by the mlr3learners package.\n\nlibrary(mlr3learners)\n\nlearner = lrn(\"classif.log_reg\")\n\nTo evaluate the predictive performance, we choose a 3-fold cross-validation and the classification error as the measure.\n\nresampling = rsmp(\"cv\", folds = 3)\nmeasure = msr(\"classif.ce\")\n\nresampling$instantiate(task)" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#classes", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#classes", - "title": "Feature Selection on the Titanic Data Set", - "section": "Classes", - "text": "Classes\nThe FSelectInstanceSingleCrit class specifies a general feature selection scenario. It includes the ObjectiveFSelect object that encodes the black box objective function which is optimized by a feature selection algorithm. The evaluated feature sets are stored in an ArchiveFSelect object. The archive provides a method for querying the best performing feature set.\nThe Terminator classes determine when to stop the feature selection. In this example we choose a terminator that stops the feature selection after 10 seconds. The sugar functions trm() and trms() can be used to retrieve terminators from the mlr_terminators dictionary.\n\nterminator = trm(\"run_time\", secs = 10)\nFSelectInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n terminator = terminator)\n\n\n* State: Not optimized\n* Objective: \n* Search Space:\n id class lower upper nlevels\n1: age ParamLgl NA NA 2\n2: embarked ParamLgl NA NA 2\n3: fare ParamLgl NA NA 2\n4: parch ParamLgl NA NA 2\n5: pclass ParamLgl NA NA 2\n6: sex ParamLgl NA NA 2\n7: sib_sp ParamLgl NA NA 2\n* Terminator: \n\n\nThe FSelector subclasses describe the feature selection strategy. The sugar function fs() can be used to retrieve feature selection algorithms from the mlr_fselectors dictionary.\n\nmlr_fselectors\n\n with 7 stored values\nKeys: design_points, exhaustive_search, genetic_search, random_search, rfe, sequential,\n shadow_variable_search" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#random-search", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#random-search", - "title": "Feature Selection on the Titanic Data Set", - "section": "Random search", - "text": "Random search\nRandom search randomly draws feature sets and evaluates them in batches. We retrieve the FSelectorRandomSearch class with the fs() sugar function and choose TerminatorEvals. We set the n_evals parameter to 10 which means that 10 feature sets are evaluated.\n\nterminator = trm(\"evals\", n_evals = 10)\ninstance = FSelectInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n terminator = terminator)\nfselector = fs(\"random_search\", batch_size = 5)\n\nThe feature selection is started by passing the FSelectInstanceSingleCrit object to the $optimize() method of FSelectorRandomSearch which generates the feature sets. These features set are internally passed to the $eval_batch() method of FSelectInstanceSingleCrit which evaluates them with the objective function and stores the results in the archive. This general interaction between the objects of mlr3fselect stays the same for the different feature selection methods. However, the way how new feature sets are generated differs depending on the chosen FSelector subclass.\n\nfselector$optimize(instance)\n\n age embarked fare parch pclass sex sib_sp features classif.ce\n1: TRUE FALSE TRUE TRUE TRUE TRUE TRUE age,fare,parch,pclass,sex,sib_sp 0.2020202\n\n\nThe ArchiveFSelect stores a data.table::data.table() which consists of the evaluated feature sets and the corresponding estimated predictive performances.\n\nas.data.table(instance$archive)\n\n\n\n\n\n \n \n age \n embarked \n fare \n parch \n pclass \n sex \n sib_sp \n classif.ce \n timestamp \n batch_nr \n \n \n\n \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n 0.2031425 \n 2022-09-17 18:34:47 \n 1 \n \n \n TRUE \n FALSE \n FALSE \n FALSE \n FALSE \n FALSE \n TRUE \n 0.3838384 \n 2022-09-17 18:34:47 \n 1 \n \n \n FALSE \n FALSE \n FALSE \n TRUE \n FALSE \n FALSE \n TRUE \n 0.3804714 \n 2022-09-17 18:34:47 \n 1 \n \n \n FALSE \n FALSE \n TRUE \n FALSE \n FALSE \n FALSE \n FALSE \n 0.3288440 \n 2022-09-17 18:34:47 \n 1 \n \n \n FALSE \n FALSE \n TRUE \n FALSE \n FALSE \n TRUE \n FALSE \n 0.2188552 \n 2022-09-17 18:34:47 \n 1 \n \n \n FALSE \n FALSE \n FALSE \n FALSE \n TRUE \n FALSE \n FALSE \n 0.3209877 \n 2022-09-17 18:34:47 \n 2 \n \n \n TRUE \n FALSE \n FALSE \n FALSE \n FALSE \n FALSE \n TRUE \n 0.3838384 \n 2022-09-17 18:34:47 \n 2 \n \n \n TRUE \n FALSE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n 0.2020202 \n 2022-09-17 18:34:47 \n 2 \n \n \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n 0.2031425 \n 2022-09-17 18:34:47 \n 2 \n \n \n TRUE \n FALSE \n TRUE \n TRUE \n FALSE \n FALSE \n FALSE \n 0.3389450 \n 2022-09-17 18:34:47 \n 2 \n \n\n\n\n\nThe associated resampling iterations can be accessed in the BenchmarkResult by calling\n\ninstance$archive$benchmark_result\n\n of 30 rows with 10 resampling runs\n nr task_id learner_id resampling_id iters warnings errors\n 1 titanic select.classif.log_reg cv 3 0 0\n 2 titanic select.classif.log_reg cv 3 0 0\n 3 titanic select.classif.log_reg cv 3 0 0\n 4 titanic select.classif.log_reg cv 3 0 0\n 5 titanic select.classif.log_reg cv 3 0 0\n 6 titanic select.classif.log_reg cv 3 0 0\n 7 titanic select.classif.log_reg cv 3 0 0\n 8 titanic select.classif.log_reg cv 3 0 0\n 9 titanic select.classif.log_reg cv 3 0 0\n 10 titanic select.classif.log_reg cv 3 0 0\n\n\nWe retrieve the best performing feature set with\n\ninstance$result\n\n age embarked fare parch pclass sex sib_sp features classif.ce\n1: TRUE FALSE TRUE TRUE TRUE TRUE TRUE age,fare,parch,pclass,sex,sib_sp 0.2020202" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#sequential-forward-selection", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#sequential-forward-selection", - "title": "Feature Selection on the Titanic Data Set", - "section": "Sequential forward selection", - "text": "Sequential forward selection\nWe try sequential forward selection. We chose TerminatorStagnation that stops the feature selection if the predictive performance does not increase anymore.\n\nterminator = trm(\"stagnation\", iters = 5)\ninstance = FSelectInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n terminator = terminator)\n\nfselector = fs(\"sequential\")\nfselector$optimize(instance)\n\n age embarked fare parch pclass sex sib_sp features classif.ce\n1: FALSE FALSE FALSE TRUE TRUE TRUE TRUE parch,pclass,sex,sib_sp 0.1964085\n\n\nThe FSelectorSequential object has a special method for displaying the optimization path of the sequential feature selection.\n\nfselector$optimization_path(instance)\n\n age embarked fare parch pclass sex sib_sp classif.ce batch_nr\n1: TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.3838384 1\n2: TRUE FALSE FALSE FALSE FALSE TRUE FALSE 0.2132435 2\n3: TRUE FALSE FALSE FALSE FALSE TRUE TRUE 0.2087542 3\n4: TRUE FALSE FALSE FALSE TRUE TRUE TRUE 0.2143659 4\n5: TRUE FALSE FALSE TRUE TRUE TRUE TRUE 0.2065095 5\n6: TRUE FALSE TRUE TRUE TRUE TRUE TRUE 0.2020202 6" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#recursive-feature-elimination", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#recursive-feature-elimination", - "title": "Feature Selection on the Titanic Data Set", - "section": "Recursive feature elimination", - "text": "Recursive feature elimination\nRecursive feature elimination utilizes the $importance() method of learners. In each iteration the feature(s) with the lowest importance score is dropped. We choose the non-recursive algorithm (recursive = FALSE) which calculates the feature importance once on the complete feature set. The recursive version (recursive = TRUE) recomputes the feature importance on the reduced feature set in every iteration.\n\nlearner = lrn(\"classif.ranger\", importance = \"impurity\")\nterminator = trm(\"none\")\ninstance = FSelectInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n terminator = terminator,\n store_models = TRUE)\n\nfselector = fs(\"rfe\", recursive = FALSE)\nfselector$optimize(instance)\n\n age embarked fare parch pclass sex sib_sp features classif.ce\n1: TRUE TRUE TRUE TRUE TRUE TRUE TRUE age,embarked,fare,parch,pclass,sex,... 0.1762065\n\n\nWe access the results.\n\nas.data.table(instance$archive, exclude_columns = c(\"runtime_learners\", \"timestamp\", \"batch_nr\", \"resample_result\", \"uhash\"))\n\n\n\n\n\n \n \n age \n embarked \n fare \n parch \n pclass \n sex \n sib_sp \n classif.ce \n importance \n \n \n\n \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n TRUE \n 0.1762065 \n 68.226237, 46.612874, 37.573185, 22.974043, 11.928321, 9.401151, 8.395001 \n \n \n TRUE \n FALSE \n TRUE \n FALSE \n FALSE \n TRUE \n FALSE \n 0.2143659 \n 68.22624, 46.61287, 37.57319" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#nested-resampling", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#nested-resampling", - "title": "Feature Selection on the Titanic Data Set", - "section": "Nested resampling", - "text": "Nested resampling\nIt is a common mistake to report the predictive performance estimated on resampling sets during the feature selection as the performance that can be expected from the combined feature selection and model training. The repeated evaluation of the model might leak information about the test sets into the model and thus leads to over-fitting and over-optimistic performance results. Nested resampling uses an outer and inner resampling to separate the feature selection from the performance estimation of the model. We can use the AutoFSelector class for running nested resampling. The AutoFSelector essentially combines a given Learner and feature selection method into a Learner with internal automatic feature selection. The inner resampling loop that is used to determine the best feature set is conducted internally each time the AutoFSelector Learner object is trained.\n\nresampling_inner = rsmp(\"cv\", folds = 5)\nmeasure = msr(\"classif.ce\")\n\nat = AutoFSelector$new(\n learner = learner,\n resampling = resampling_inner,\n measure = measure,\n terminator = terminator,\n fselect = fs(\"sequential\"),\n store_models = TRUE)\n\nWe put the AutoFSelector into a resample() call to get the outer resampling loop.\n\nresampling_outer = rsmp(\"cv\", folds = 3)\n\nrr = resample(task, at, resampling_outer, store_models = TRUE)\n\n\n\n\n\n\n\nThe aggregated performance of all outer resampling iterations is the unbiased predictive performance we can expected from the logistic regression model with an optimized feature set found by sequential selection.\n\nrr$aggregate()\n\nclassif.ce \n 0.1907969 \n\n\nWe check whether the feature sets that were selected in the inner resampling are stable. The selected feature sets should not differ too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduces too much randomness. Usually, we aim for the selection of similar feature sets for all outer training sets.\n\nextract_inner_fselect_results(rr)\n\n\n\n\n\n\n\n\nNext, we want to compare the predictive performances estimated on the outer resampling to the inner resampling. Significantly lower predictive performances on the outer resampling indicate that the models with the optimized feature sets overfit the data.\n\nrr$score()\n\n\n\n\n\n \n \n iteration \n task_id \n learner_id \n resampling_id \n classif.ce \n \n \n\n \n 1 \n titanic \n classif.ranger.fselector \n cv \n 0.2289562 \n \n \n 2 \n titanic \n classif.ranger.fselector \n cv \n 0.1784512 \n \n \n 3 \n titanic \n classif.ranger.fselector \n cv \n 0.1649832 \n \n\n\n\n\nThe archives of the AutoFSelectors gives us all evaluated feature sets with the associated predictive performances.\n\nextract_inner_fselect_archives(rr)" - }, - { - "objectID": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#shortcuts", - "href": "gallery/2020-09-14-mlr3fselect-basic/mlr3fselect-basic.html#shortcuts", - "title": "Feature Selection on the Titanic Data Set", - "section": "Shortcuts", - "text": "Shortcuts\nSelecting a feature subset can be shortened by using the fselect()-shortcut.\n\ninstance = fselect(\n method = \"random_search\",\n task = tsk(\"iris\"),\n learner = lrn(\"classif.log_reg\"),\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 10\n)\n\nApplying nested resampling can be shortened by using the fselect_nested()-shortcut.\n\nrr = fselect_nested(\n method = \"random_search\",\n task = tsk(\"iris\"),\n learner = lrn(\"classif.log_reg\"),\n inner_resampling = rsmp (\"cv\", folds = 3),\n outer_resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 10\n)" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html", - "title": "Impute Missing Variables", - "section": "", - "text": "This tutorial assumes familiarity with the basics of mlr3pipelines. Consult the mlr3book if some aspects are not fully understandable. It deals with the problem of missing data.\nThe random forest implementation in the package ranger unfortunately does not support missing values. Therefore, it is required to impute missing features before passing the data to the learner.\nWe show how to use mlr3pipelines to augment the ranger learner with automatic imputation.\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nLoading required package: mlr3\n\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#construct-the-base-objects", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#construct-the-base-objects", - "title": "Impute Missing Variables", - "section": "Construct the Base Objects", - "text": "Construct the Base Objects\nFirst, we take an example task with missing values (pima) and create the ranger learner:\n\nlibrary(mlr3learners)\n\ntask = tsk(\"pima\")\nprint(task)\n\n (768 x 9): Pima Indian Diabetes\n* Target: diabetes\n* Properties: twoclass\n* Features (8):\n - dbl (8): age, glucose, insulin, mass, pedigree, pregnant, pressure, triceps\n\nlearner = lrn(\"classif.ranger\")\nprint(learner)\n\n\n* Model: -\n* Parameters: num.threads=1\n* Packages: mlr3, mlr3learners, ranger\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, character, factor, ordered\n* Properties: hotstart_backward, importance, multiclass, oob_error, twoclass, weights\n\n\nWe can now inspect the task for missing values. task$missings() returns the count of missing values for each variable.\n\ntask$missings()\n\ndiabetes age glucose insulin mass pedigree pregnant pressure triceps \n 0 0 5 374 11 0 0 35 227 \n\n\nAdditionally, we can see that the ranger learner can not handle missing values:\n\nlearner$properties\n\n[1] \"hotstart_backward\" \"importance\" \"multiclass\" \"oob_error\" \"twoclass\" \n[6] \"weights\" \n\n\nFor comparison, other learners, e.g. the rpart learner can handle missing values internally.\n\nlrn(\"classif.rpart\")$properties\n\n[1] \"importance\" \"missings\" \"multiclass\" \"selected_features\" \"twoclass\" \n[6] \"weights\" \n\n\nBefore we dive deeper, we quickly try to visualize the columns with many missing values:\n\nautoplot(task$clone()$select(c(\"insulin\", \"triceps\")), type = \"pairs\")" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#operators-overview", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#operators-overview", - "title": "Impute Missing Variables", - "section": "Operators overview", - "text": "Operators overview\nAn overview over implemented PipeOps for imputation can be obtained like so:\n\nas.data.table(mlr_pipeops)[tags %in% \"missings\", list(key)]\n\n key\n1: imputeconstant\n2: imputehist\n3: imputelearner\n4: imputemean\n5: imputemedian\n6: imputemode\n7: imputeoor\n8: imputesample" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#construct-operators", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#construct-operators", - "title": "Impute Missing Variables", - "section": "Construct Operators", - "text": "Construct Operators\nmlr3pipelines contains several imputation methods. We focus on rather simple ones, and show how to impute missing values for factor features and numeric features respectively.\nSince our task only has numeric features, we do not need to deal with imputing factor levels, and can instead concentrate on imputing numeric values:\nWe do this in a two-step process: * We create new indicator columns, that tells us whether the value of a feature is “missing” or “present”. We achieve this using the missind PipeOp.\n\nAfterwards, we impute every missing value by sampling from the histogram of the respective column. We achieve this using the imputehist PipeOp.\n\nWe also have to make sure to apply the pipe operators in the correct order!\n\nimp_missind = po(\"missind\")\nimp_num = po(\"imputehist\", affect_columns = selector_type(\"numeric\"))\n\nIn order to better understand we can look at the results of every PipeOp separately.\nWe can manually trigger the PipeOp to test the operator on our task:\n\ntask_ext = imp_missind$train(list(task))[[1]]\ntask_ext$data()\n\n diabetes missing_glucose missing_insulin missing_mass missing_pressure missing_triceps\n 1: pos present missing present present present\n 2: neg present missing present present present\n 3: pos present missing present present missing\n 4: neg present present present present present\n 5: pos present present present present present\n --- \n764: neg present present present present present\n765: neg present missing present present present\n766: neg present present present present present\n767: pos present missing present present missing\n768: neg present missing present present present\n\n\nFor imputehist, we can do the same:\n\ntask_ext = imp_num$train(list(task))[[1]]\ntask_ext$data()\n\n diabetes age pedigree pregnant glucose insulin mass pressure triceps\n 1: pos 50 0.627 6 148 163.11747 33.6 72 35.000000\n 2: neg 31 0.351 1 85 160.63628 26.6 66 29.000000\n 3: pos 32 0.672 8 183 297.18282 23.3 64 8.204983\n 4: neg 21 0.167 1 89 94.00000 28.1 66 23.000000\n 5: pos 33 2.288 0 137 168.00000 43.1 40 35.000000\n --- \n764: neg 63 0.171 10 101 180.00000 32.9 76 48.000000\n765: neg 27 0.340 2 122 83.69836 36.8 70 27.000000\n766: neg 30 0.245 5 121 112.00000 26.2 72 23.000000\n767: pos 47 0.349 1 126 68.49318 30.1 60 24.460702\n768: neg 23 0.315 1 93 17.80534 30.4 70 31.000000\n\n\nThis time we obtain the imputed data set without missing values.\n\ntask_ext$missings()\n\ndiabetes age pedigree pregnant glucose insulin mass pressure triceps \n 0 0 0 0 0 0 0 0 0" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#putting-everything-together", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#putting-everything-together", - "title": "Impute Missing Variables", - "section": "Putting everything together", - "text": "Putting everything together\nNow we have to put all PipeOps together in order to form a graph that handles imputation automatically.\nWe do this by creating a Graph that copies the data twice, processes each copy using the respective imputation method and afterwards unions the features. For this we need the following two PipeOps : * copy: Creates copies of the data. * featureunion Merges the two tasks together.\n\ngraph = po(\"copy\", 2) %>>%\n gunion(list(imp_missind, imp_num)) %>>%\n po(\"featureunion\")\n\nas a last step we append the learner we planned on using:\n\ngraph = graph %>>% po(learner)\n\nWe can now visualize the resulting graph:\n\ngraph$plot()" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#resampling", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#resampling", - "title": "Impute Missing Variables", - "section": "Resampling", - "text": "Resampling\nCorrect imputation is especially important when applying imputation to held-out data during the predict step. If applied incorrectly, imputation could leak info from the test set, which potentially skews our performance estimates. mlr3pipelines takes this complexity away from the user and handles correct imputation internally.\nBy wrapping this graph into a GraphLearner, we can now train resample the full graph, here with a 3-fold cross validation:\n\ngraph_learner = as_learner(graph)\nrr = resample(task, graph_learner, rsmp(\"cv\", folds = 3))\nrr$aggregate()\n\nclassif.ce \n 0.2421875" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#missing-values-during-prediction", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#missing-values-during-prediction", - "title": "Impute Missing Variables", - "section": "Missing values during prediction", - "text": "Missing values during prediction\nIn some cases, we have missing values only in the data we want to predict on. In order to showcase this, we create a copy of the task with several more missing columns.\n\ndt = task$data()\ndt[1:10, \"age\"] = NA\ndt[30:70, \"pedigree\"] = NA\ntask_2 = as_task_classif(dt, id = \"pima2\", target = \"diabetes\")\n\nAnd now we learn on task, while trying to predict on task_2.\n\ngraph_learner$train(task)\ngraph_learner$predict(task_2)\n\n for 768 observations:\n row_ids truth response\n 1 pos neg\n 2 neg neg\n 3 pos pos\n--- \n 766 neg neg\n 767 pos pos\n 768 neg neg" - }, - { - "objectID": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#missing-factor-features", - "href": "gallery/2020-01-30-impute-missing-levels/2020-01-30-impute-missing-levels.html#missing-factor-features", - "title": "Impute Missing Variables", - "section": "Missing factor features", - "text": "Missing factor features\nFor factor features, the process works analogously. Instead of using imputehist, we can for example use imputeoor. This will simply replace every NA in each factor variable with a new value missing.\nA full graph might the look like this:\n\nimp_missind = po(\"missind\", affect_columns = NULL, which = \"all\")\nimp_fct = po(\"imputeoor\", affect_columns = selector_type(\"factor\"))\ngraph = po(\"copy\", 2) %>>%\n gunion(list(imp_missind, imp_num %>>% imp_fct)) %>>%\n po(\"featureunion\")\n\nNote that we specify the parameter affect_columns = NULL when initializing missind, because we also want indicator columns for our factor features. By default, affect_columns would be set to selector_invert(selector_type(c(\"factor\", \"ordered\", \"character\"))). We also set the parameter which to \"all\" to add indicator columns for all features, regardless whether values were missing during training or not.\nIn order to test out our new graph, we again create a situation where our task has missing factor levels. As the (pima) task does not have any factor levels, we use the famous (boston_housing) task.\n\n# task_bh_1 is the training data without missings\ntask_bh_1 = tsk(\"boston_housing\")\n\n# task_bh_2 is the prediction data with missings\ndt = task_bh_1$data()\ndt[1:10, chas := NA][20:30, rm := NA]\ntask_bh_2 = as_task_regr(dt, id = \"bh\", target = \"medv\")\n\nNow we train on task_bh_1 and predict on task_bh_2:\n\ngraph_learner = as_learner(graph %>>% po(lrn(\"regr.ranger\")))\ngraph_learner$train(task_bh_1)\ngraph_learner$predict(task_bh_2)\n\n for 506 observations:\n row_ids truth response\n 1 24.0 24.67359\n 2 21.6 22.34092\n 3 34.7 33.77385\n--- \n 504 23.9 23.76018\n 505 22.0 22.50533\n 506 11.9 16.15236\n\n\nSuccess! We learned how to deal with missing values in less than 10 minutes." - }, - { - "objectID": "gallery/2020-06-15-target-transformations-via-pipelines/target-transformations-via-pipelines.html", - "href": "gallery/2020-06-15-target-transformations-via-pipelines/target-transformations-via-pipelines.html", - "title": "Target Transformations via Pipelines", - "section": "", - "text": "Transforming the target variable often can lead to predictive improvement and is a widely used tool. Typical transformations are for example the \\(\\log\\) transformation of the target aiming at minimizing (right) skewness, or the Box Cox and Yeo-Johnson transformations being more flexible but having a similar goal.\nOne option to perform, e.g., a \\(\\log\\) transformation would be to manually transform the target prior to training a Learner (and also predicting from it) and then manually invert this transformation via \\(\\exp\\) after predicting from the Learner. This is quite cumbersome, especially if a transformation and inverse transformation require information about both the training and prediction data.\nIn this post, we show how to do various kinds of target transformations using mlr3pipelines and explain the design of the target transformation and inversion PipeOps.\nYou will:\n\nlearn how to do simple target transformations using PipeOpTargetMutate\nbe introduced to the abstract base class to implement custom target transformations, PipeOpTargetTrafo\nimplement a custom target transformation PipeOp, PipeOpTargetTrafoBoxCox\n\nAs a prerequisite, you should be quite familiar with mlr3pipelines, i.e, know about the $state field of PipeOps, input and output channels, as well as Graphs. We will start with a PipeOp for simple target transformations, PipeOpTargetMutate.\nWe load the most important packages for this example.\n\nlibrary(mlr3)\nlibrary(mlr3learners)\nlibrary(mlr3pipelines)\nlibrary(paradox)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\n\nIn all sections we will use the mtcars regression task with mpg being a numerical, positive target:\n\ntask = tsk(\"mtcars\")\nsummary(task$data(cols = task$target_names))\n\n mpg \n Min. :10.40 \n 1st Qu.:15.43 \n Median :19.20 \n Mean :20.09 \n 3rd Qu.:22.80 \n Max. :33.90 \n\n\nMoreover, as a Learner we will use an ordinary linear regression learner:\n\nlearner_lm = lrn(\"regr.lm\")\n\n\nSimple Target Transformations\nThe term simple refers to transformations that are given by a function of the target, relying on no other arguments (constants are of course allowed). The most prominent example is given by the \\(\\log\\) transformation which we can later invert by applying the \\(\\exp\\) transformation.\nIf you are only interested in doing such a transformation and you do not have the time to read more of this post, simply use the following syntactic sugar:\n\ng_ppl = ppl(\"targettrafo\", graph = learner_lm)\ng_ppl$param_set$values$targetmutate.trafo = function(x) log(x)\ng_ppl$param_set$values$targetmutate.inverter = function(x) list(response = exp(x$response))\n\nThis constructs a Graph that will \\(\\log\\) transform the target prior to training the linear regression learner (or predicting from it) and \\(\\exp\\) transform the target after predicting from it. Note that you can supply any other Learner or even a whole Graph as the graph argument.\nNow, we will go into more detail about how this actually works:\nWe can perform a \\(\\log\\) transformation of our numerical, positive target, mpg, using PipeOpTargetMutate (by default, ppl(\"targettrafo\") uses this target transformation PipeOp):\n\ntrafo = po(\"targetmutate\", param_vals = list(trafo = function(x) log(x)))\n\nWe have to specify the trafo parameter as a function of x (which will internally be evaluated to be the target of the Task): trafo = function(x) log(x)). In principle, this is all that is needed to transform the target prior to training a Learner (or predicting from it), i.e., if we now train this PipeOp, we see that the target is transformed as specified:\n\ntrafo$train(list(task))$output$data(cols = task$target_names)\n\n mpg\n 1: 3.044522\n 2: 3.044522\n 3: 3.126761\n 4: 3.063391\n 5: 2.928524\n 6: 2.895912\n 7: 2.660260\n 8: 3.194583\n 9: 3.126761\n10: 2.954910\n11: 2.879198\n12: 2.797281\n13: 2.850707\n14: 2.721295\n15: 2.341806\n16: 2.341806\n17: 2.687847\n18: 3.478158\n19: 3.414443\n20: 3.523415\n21: 3.068053\n22: 2.740840\n23: 2.721295\n24: 2.587764\n25: 2.954910\n26: 3.306887\n27: 3.258097\n28: 3.414443\n29: 2.760010\n30: 2.980619\n31: 2.708050\n32: 3.063391\n mpg\n\n\nAfter having predicted from the Learner we could then proceed to apply the inverse transformation function in a similar manner. However, in mlr3pipelines, we decided to go with a more unified design of handling target transformations. In all target transformation PipeOps also the inverse transformation function of the target has to be specified. Therefore, in PipeOpTargetMutate, the parameter inverter also has to be correctly specified:\n\ntrafo$param_set$values$inverter = function(x) list(response = exp(x$response))\n\nInternally, this function will be applied to the data.table downstream of a Prediction object without the $row_id and $truth columns, and we specify that the $response column should be transformed. Note that applying the inverse transformation will typically only be done to the $response column, because transforming standard errors or probabilities is often not straightforward.\nTo actually carry out the inverse transformation function after predicting from the Learner, we then rely on PipeOpTargetInvert. PipeOpTargetInvert has an empty ParamSet and its sole purpose is to apply the inverse transformation function after having predicted from a Learner (note that this whole design of target transformations may seem somewhat over-engineered at first glance, however, we will learn of its advantages when we later move to the advanced section).\nPipeOpTargetInvert has two input channels named \"fun\" and \"prediction\". During training, both take NULL as input (because this is what a Learner’s \"output\" output and PipeOpTargetMutate’s \"fun\" output will return during training). During prediction, the \"prediction\" input takes a Prediction, and the \"fun\" input takes the \"fun\" output from PipeOpTargetMutate (you may have noticed already, that PipeOpTargetMutate has actually two outputs, \"fun\" and \"output\", with \"fun\" returning NULL during training and a function during prediction, while \"output\" always returns the transformed input Task). We can see this, if we look at:\n\ntrafo$output\n\n name train predict\n1: fun NULL function\n2: output Task Task\n\ntrafo$predict(list(task))\n\n$fun\nfunction (inputs) \n{\n assert_list(inputs, len = 1L, types = \"Prediction\")\n list(private$.invert(inputs[[1L]], predict_phase_state))\n}\n\n\n\n$output\n (32 x 11): Motor Trends\n* Target: mpg\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\n\nWe will talk more about such technical details in the advanced section. For now, to finally construct our target transformation pipeline, we build a Graph:\n\ng = Graph$new()\ng$add_pipeop(trafo)\ng$add_pipeop(learner_lm)\ng$add_pipeop(po(\"targetinvert\"))\n\nManually connecting the edges is quite cumbersome. First we connect the \"output\" output of \"targetmutate\" to the \"input\" input of \"regr.lm\":\n\ng$add_edge(src_id = \"targetmutate\", dst_id = \"regr.lm\",\n src_channel = 2, dst_channel = 1)\n\nThen we connect the \"output\" output of \"regr.lm\" to the \"prediction\" input of \"targetinvert\":\n\ng$add_edge(src_id = \"regr.lm\", dst_id = \"targetinvert\",\n src_channel = 1, dst_channel = 2)\n\nFinally, we connect the \"fun\" output of \"targetmutate\" to the \"fun\" input of \"targetinvert\":\n\ng$add_edge(src_id = \"targetmutate\", dst_id = \"targetinvert\",\n src_channel = 1, dst_channel = 1)\n\nThis graph (which is conceptually the same graph as constructed via the ppl(\"targettrafo\") syntactic sugar above) looks like the following:\n\ng$plot(html = TRUE)\n\n\n\n\n\nWe can then finally call $train() and $predict() (prior to this we wrap the Graph in a GraphLearner):\n\ngl = GraphLearner$new(g)\ngl$train(task)\ngl$state\n\n$model\n$model$targetmutate\nlist()\n\n$model$regr.lm\n$model$regr.lm$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n(Intercept) am carb cyl disp drat gear hp qsec \n 2.776e+00 4.738e-02 -2.012e-02 7.657e-03 4.989e-05 2.220e-02 5.925e-02 -8.964e-04 3.077e-02 \n vs wt \n -2.874e-03 -1.723e-01 \n\n\n$model$regr.lm$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$model$regr.lm$train_time\n[1] 0.004\n\n$model$regr.lm$param_vals\nnamed list()\n\n$model$regr.lm$task_hash\n[1] \"35f61536974e33c0\"\n\n$model$regr.lm$data_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$model$regr.lm$task_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$model$regr.lm$mlr3_version\n[1] '0.14.0'\n\n$model$regr.lm$train_task\n (32 x 11): Motor Trends\n* Target: mpg\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\n\n$model$targetinvert\nlist()\n\n\n$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$train_time\n[1] 0.024\n\n$param_vals\n$param_vals$targetmutate.trafo\nfunction(x) log(x)\n\n\n$param_vals$targetmutate.inverter\nfunction(x) list(response = exp(x$response))\n\n\n$task_hash\n[1] \"61fc311c6e5b602b\"\n\n$data_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$task_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$mlr3_version\n[1] '0.14.0'\n\n$train_task\n (32 x 11): Motor Trends\n* Target: mpg\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\ngl$predict(task)\n\n for 32 observations:\n row_ids truth response\n 1 21.0 21.67976\n 2 21.0 21.10831\n 3 22.8 25.73690\n--- \n 30 19.7 19.58533\n 31 15.0 14.11015\n 32 21.4 23.11105\n\n\nand contrast this with $train() and $predict() of the naive linear regression learner (also look at the estimated coefficients of the linear regression contained in $state$model):\n\nlearner_lm$train(task)\nlearner_lm$state\n\n$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n(Intercept) am carb cyl disp drat gear hp qsec \n 12.30337 2.52023 -0.19942 -0.11144 0.01334 0.78711 0.65541 -0.02148 0.82104 \n vs wt \n 0.31776 -3.71530 \n\n\n$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$train_time\n[1] 0.001\n\n$param_vals\nnamed list()\n\n$task_hash\n[1] \"61fc311c6e5b602b\"\n\n$data_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$task_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$mlr3_version\n[1] '0.14.0'\n\n$train_task\n (32 x 11): Motor Trends\n* Target: mpg\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\nlearner_lm$predict(task)\n\n for 32 observations:\n row_ids truth response\n 1 21.0 22.59951\n 2 21.0 22.11189\n 3 22.8 26.25064\n--- \n 30 19.7 19.69383\n 31 15.0 13.94112\n 32 21.4 24.36827\n\n\nYou should continue reading, if you are interested in more advanced target transformations, i.e., where the transformation and inverse transformation require information about both the training and prediction data.\nFirst we will introduce the abstract base class for doing target transformations, PipeOpTargetTrafo, from which PipeOpTargetMutate inherits.\n\n\nAbstract Base Class: PipeOpTargetTrafo\nNo matter how “complicated” the actual target transformation and inverse transformation may be, applying the inverse transformation function after having predicted from a Learner will always be done via PipeOpTargetInvert (as already outlined above, PipeOpTargetInvert has an empty ParamSet and its sole purpose is to apply the inverse transformation function after having predicted from a Learner). All Graphs for doing target transformations will therefore look similar like the simple one above, i.e., a target transformation PipeOp followed by some Learner or a whole Graph, followed by PipeOpTargetInvert. Therefore, using ppl(\"targettrafo\") to construct such Graphs is highly recommended.\nTo allow for more advanced target transformations, we now have a closer look at the abstract base class, PipeOpTargetTrafo:\nPipeOpTargetTrafo has one input channel, named \"input\" taking a Task both during training and prediction. It’s two output channels are named \"fun\" and \"output\". During training \"fun\" returns NULL and during prediction \"fun\" returns a function that will be used by PipeOpTargetInvert to perform the inverse target transformation on PipeOpTargetInvert’s \"prediction\" input. \"output\" returns the modified input Task both during training and prediction.\nSubclasses can overload up to four functions:\n\n.get_state() takes the input Task and returns a list() which will internally be used to set the $state. Typically it is sensible to make use of the $state during .transform() and .train_invert(). The base implementation returns list() and should be overloaded if setting the state is desired.\n.transform() takes the input Task and returns a modified Task (i.e., the Task with the transformed target). This is the main function for doing the actual target transformation. Note that .get_state() is evaluated a single time during training right before .transform() and therefore, you can rely on the $state that has been set. To update the input Task with respect to the transformed target, subclasses should make use of the convert_task() function and drop the original target from the Task. .transform() also accepts a phase argument that will receive \"train\" during training and \"predict\" during prediction. This can be used to enable different behavior during training and prediction. .transform() should always be overloaded by subclasses.\n.train_invert() takes the input Task and returns a predict_phase_state object. This can be anything. Note that .train_invert() should not modify the input Task. The base implementation returns a list with a single argument, the $truth column of the input Task and should be overloaded if a more training-phase-dependent state is desired.\n.invert() takes a Prediction and a predict_phase_state object as inputs and returns a Prediction. This is the main function for specifying the actual inverse target transformation that will later be carried out by PipeOpTargetInvert. Internally a private helper function , .invert_help() will construct the function that will be returned by the \"fun\" output of PipeOpTargetTrafo so that PipeOpTargetInvert can later simply dispatch this inverse target transformation on its \"prediction\" input.\n\nThe supposed workflow of a class inherited from PipeOpTargetTrafo is given in the following figure:\n\n\n\n\n\n\n\n\n\nTo solidify our understanding we will design a new target transformation PipeOp in the next section: PipeOpTargetTrafoBoxCox\n\n\nHands on: PipeOpTargetTrafoBoxCox\n\nlibrary(R6)\n\nThe Box-Cox transformation of a target \\(y_{i}\\) is given as:\n\\[y_{i}(\\lambda) = \\begin{cases}\n\\frac{y_{i}^{\\lambda} - 1}{\\lambda} & \\text{if}~\\lambda \\neq 0; \\\\\n\\log(y_{i}) & \\text{if}~\\lambda = 0\n\\end{cases}\\]\nmlr3pipelines already supports the Box-Cox transformation for numerical, positive features, see ?PipeOpBoxCox.\nHere we will design a PipeOp to apply the Box-Cox transformation as a target transformation. The \\(\\lambda\\) parameter of the transformation is estimated during training and used for both the training and prediction transformation. After predicting from a Learner we will as always apply the inverse transformation function. To do the actual transformation we will use bestNormalize::boxcox().\nFirst, we inherit from PipeOpTargetTrafo and overload the initialize() function:\n\nPipeOpTargetTrafoBoxCox = R6Class(\"PipeOpTargetTrafoBoxCox\",\n inherit = PipeOpTargetTrafo,\n public = list(\n initialize = function(id = \"targettrafoboxcox\", param_vals = list()) {\n param_set = ps(\n standardize = p_lgl(default = TRUE, tags = c(\"train\", \"boxcox\")),\n eps = p_dbl(default = 0.001, lower = 0, tags = c(\"train\", \"boxcox\")),\n lower = p_dbl(default = -1L, tags = c(\"train\", \"boxcox\")),\n upper = p_dbl(default = 2L, tags = c(\"train\", \"boxcox\"))\n )\n super$initialize(id = id, param_set = param_set, param_vals = param_vals,\n packages = \"bestNormalize\", task_type_in = \"TaskRegr\",\n task_type_out = \"TaskRegr\")\n }\n ),\n private = list(\n\n .get_state = function(task) {\n ...\n },\n\n .transform = function(task, phase) {\n ...\n },\n\n .train_invert = function(task) {\n ...\n },\n\n .invert = function(prediction, predict_phase_state) {\n ...\n }\n )\n)\n\nAs parameters, we allow \"standardize\" (whether to center and scale the transformed values to attempt a standard normal distribution), \"eps\" (tolerance parameter to identify if the \\(\\lambda\\) parameter is equal to zero), \"lower\" (lower value for the estimation of the \\(\\lambda\\) parameter) and \"upper\" (upper value for the estimation of the \\(\\lambda\\) parameter). Note that we set task_type_in = \"TaskRegr\" and task_type_out = \"TaskRegr\" to specify that this PipeOp only works for regression Tasks.\nSecond, we overload the four functions as mentioned above.\nWe start with .get_state(). We extract the target and apply the Box-Cox transformation to the target. This yields an object of class \"boxcox\" which we will wrap in a list() and set as the $state (bc$x.t = NULL and bc$x = NULL is done to save some memory because we do not need the transformed original data and original data later):\n\n .get_state = function(task) {\n target = task$data(cols = task$target_names)[[1L]]\n bc = mlr3misc::invoke(bestNormalize::boxcox, target,\n .args = self$param_set$get_values(tags = \"boxcox\"))\n bc$x.t = NULL\n bc$x = NULL\n list(bc = bc)\n },\n\nNext, we tackle .transform(). This is quite straightforward, because objects of class \"boxcox\" have their own predict method which we can use here to carry out the actual Box-Cox transformation based on the learned \\(\\lambda\\) parameter as stored in the \"boxcox\" object in the $state (both during training and prediction). We then rename the target, add it to the task and finally update the task with respect to this new target:\n\n .transform = function(task, phase) {\n target = task$data(cols = task$target_names)[[1L]]\n new_target = as.data.table(predict(self$state$bc, newdata = target))\n colnames(new_target) = paste0(task$target_names, \".bc\")\n task$cbind(new_target)\n convert_task(task, target = colnames(new_target),\n drop_original_target = TRUE)\n },\n\nTime to overload .train_invert(). This is even more straightforward, because the prediction method for objects of class \"boxcox\" directly allows for inverting the transformation via setting the argument inverse = TRUE. Therefore, we only need the \"boxcox\" object stored in the $state along the $truth column of the input Task (remember that this list will later be available as the predict_phase_state object):\n\n .train_invert = function(task) {\n list(truth = task$truth(), bc = self$state$bc)\n },\n\nFinally, we overload .invert(). We extract the truth from the predict_phase_state and the response from the Prediction. We then apply the inverse Box-Cox transformation to the response based on the \\(\\lambda\\) parameter and the mean and standard deviation learned during training, relying on the predict_phase_state object. Finally, we construct a new Prediction object:\n\n .invert = function(prediction, predict_phase_state) {\n truth = predict_phase_state$truth\n response = predict(predict_phase_state$bc, newdata = prediction$response,\n inverse = TRUE)\n PredictionRegr$new(row_ids = prediction$row_ids, truth = truth,\n response = response)\n }\n\n\n\n\nNote that this PipeOp is ill-equipped to handle the case of predict_type = \"se\", i.e., we always only return a response prediction (as outlined above, this is the case for most target transformations, because transforming standard errors or probabilities of a prediction is often not straightforward). We could of course check whether the predict_type is set to \"se\" and if this is the case, return NA as the standard errors.\nTo construct our final target transformation Graph with our linear regression learner, we again simply make use of ppl(\"targettrafo\"):\n\ng_bc = ppl(\"targettrafo\", graph = learner_lm,\n trafo_pipeop = PipeOpTargetTrafoBoxCox$new())\n\nThe following plot should already look quite familiar:\n\ng_bc$plot(html = TRUE)\n\n\n\n\n\nFinally we $train() and $predict() on the task (again, we wrap the Graph in a GraphLearner):\n\ngl_bc = GraphLearner$new(g_bc)\ngl_bc$train(task)\ngl_bc$state\n\n$model\n$model$regr.lm\n$model$regr.lm$model\n\nCall:\nstats::lm(formula = task$formula(), data = task$data())\n\nCoefficients:\n(Intercept) am carb cyl disp drat gear hp qsec \n -0.6272999 0.1670950 -0.0663126 0.0237529 0.0002376 0.0759944 0.1963335 -0.0030367 0.1043210 \n vs wt \n -0.0080166 -0.5800635 \n\n\n$model$regr.lm$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$model$regr.lm$train_time\n[1] 0.003\n\n$model$regr.lm$param_vals\nnamed list()\n\n$model$regr.lm$task_hash\n[1] \"b0a8b06ecc566f0d\"\n\n$model$regr.lm$data_prototype\nEmpty data.table (0 rows and 11 cols): mpg.bc,am,carb,cyl,disp,drat...\n\n$model$regr.lm$task_prototype\nEmpty data.table (0 rows and 11 cols): mpg.bc,am,carb,cyl,disp,drat...\n\n$model$regr.lm$mlr3_version\n[1] '0.14.0'\n\n$model$regr.lm$train_task\n (32 x 11): Motor Trends\n* Target: mpg.bc\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\n\n$model$targettrafoboxcox\n$model$targettrafoboxcox$bc\nStandardized Box Cox Transformation with 32 nonmissing obs.:\n Estimated statistics:\n - lambda = 0.02955701 \n - mean (before standardization) = 3.092016 \n - sd (before standardization) = 0.324959 \n\n\n$model$targetinvert\nlist()\n\n\n$log\nEmpty data.table (0 rows and 3 cols): stage,class,msg\n\n$train_time\n[1] 0.024\n\n$param_vals\nnamed list()\n\n$task_hash\n[1] \"61fc311c6e5b602b\"\n\n$data_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$task_prototype\nEmpty data.table (0 rows and 11 cols): mpg,am,carb,cyl,disp,drat...\n\n$mlr3_version\n[1] '0.14.0'\n\n$train_task\n (32 x 11): Motor Trends\n* Target: mpg\n* Properties: -\n* Features (10):\n - dbl (10): am, carb, cyl, disp, drat, gear, hp, qsec, vs, wt\n\ngl_bc$predict(task)\n\n for 32 observations:\n row_ids truth response\n 1 21.0 21.70854\n 2 21.0 21.13946\n 3 22.8 25.75242\n--- \n 30 19.7 19.58934\n 31 15.0 14.10658\n 32 21.4 23.15263\n\n\nWe could now proceed to benchmark our different target transformations:\n\nbg = benchmark_grid(list(task), learners = list(learner_lm, gl, gl_bc),\n resamplings = list(rsmp(\"cv\", folds = 10)))\nbmr = benchmark(bg)\n\n\nbmr$aggregate(msr(\"regr.mse\"))\n\n nr resample_result task_id learner_id resampling_id iters regr.mse\n1: 1 mtcars regr.lm cv 10 11.866071\n2: 2 mtcars targetmutate.regr.lm.targetinvert cv 10 7.793303\n3: 3 mtcars targettrafoboxcox.regr.lm.targetinvert cv 10 8.230192" - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html", - "title": "Feature Engineering of Date-Time Variables", - "section": "", - "text": "In this tutorial, we demonstrate how mlr3pipelines can be used to easily engineer features based on date-time variables. Relying on the Bike Sharing Dataset and the ranger learner we compare the root mean square error (RMSE) of a random forest using the original features (baseline), to the RMSE of a random forest using newly engineered features on top of the original ones." - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#motivation", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#motivation", - "title": "Feature Engineering of Date-Time Variables", - "section": "Motivation", - "text": "Motivation\nA single date-time variable (i.e., a POSIXct column) contains plenty of information ranging from year, month, day, hour, minute and second to other features such as week of the year, or day of the week. Moreover, most of these features are of cyclical nature, i.e., the eleventh and twelfth hour of a day are one hour apart, but so are the 23rd hour and midnight of the other day (see also this blog post and fastai for more information).\nNot respecting this cyclical nature results in treating hours on a linear continuum. One way to handle a cyclical feature \\(\\mathbf{x}\\) is to compute the sine and cosine transformation of \\(\\frac{2 \\pi \\mathbf{x}}{\\mathbf{x}_{\\text{max}}}\\), with \\(\\mathbf{x}_{\\text{max}} = 24\\) for hours and \\(60\\) for minutes and seconds.\nThis results in a two-dimensional representation of the feature:\n\n\n\n\n\n\n\n\n\nmlr3pipelines provides the PipeOpDateFeatures pipeline which can be used to automatically engineer features based on POSIXct columns, including handling of cyclical features.\nThis is useful as most learners naturally cannot handle dates and POSIXct variables and therefore require conversion prior to training." - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#prerequisites", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#prerequisites", - "title": "Feature Engineering of Date-Time Variables", - "section": "Prerequisites", - "text": "Prerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners.\n\nlibrary(mlr3verse)\nlibrary(mlr3learners)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#bike-sharing", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#bike-sharing", - "title": "Feature Engineering of Date-Time Variables", - "section": "Bike Sharing", - "text": "Bike Sharing\nThe Bike Sharing Dataset contains the hourly count of rental bikes between years 2011 and 2012 in Capital bikeshare system with the corresponding weather and seasonal information. The dataset can be downloaded from the UCI Machine Learning Repository. After reading in the data, we fix some factor levels, and convert some data types:\nThe Bike Sharing Dataset contains the hourly count of rental bikes between years 2011 and 2012 in Capital bikeshare system with the corresponding weather and seasonal information. We load the data set from the mlr3data package.\n\ndata(\"bike_sharing\", package = \"mlr3data\")\n\nOur goal will be to predict the total number of rented bikes on a given day: cnt.\n\nskimr::skim(bike_sharing)\n\n\nData summary\n\n\nName\nbike_sharing\n\n\nNumber of rows\n17379\n\n\nNumber of columns\n14\n\n\nKey\nNULL\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\ncharacter\n1\n\n\nfactor\n2\n\n\nlogical\n2\n\n\nnumeric\n9\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: character\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmin\nmax\nempty\nn_unique\nwhitespace\n\n\n\n\ndate\n0\n1\n10\n10\n0\n731\n0\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\nseason\n0\n1\nFALSE\n4\nsum: 4496, spr: 4409, win: 4242, fal: 4232\n\n\nweather\n0\n1\nFALSE\n4\n1: 11413, 2: 4544, 3: 1419, 4: 3\n\n\n\nVariable type: logical\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\ncount\n\n\n\n\nholiday\n0\n1\n0.03\nFAL: 16879, TRU: 500\n\n\nworking_day\n0\n1\n0.68\nTRU: 11865, FAL: 5514\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nyear\n0\n1\n0.50\n0.50\n0.00\n0.00\n1.00\n1.00\n1.00\n▇▁▁▁▇\n\n\nmonth\n0\n1\n6.54\n3.44\n1.00\n4.00\n7.00\n10.00\n12.00\n▇▆▆▅▇\n\n\nhour\n0\n1\n11.55\n6.91\n0.00\n6.00\n12.00\n18.00\n23.00\n▇▇▆▇▇\n\n\nweekday\n0\n1\n3.00\n2.01\n0.00\n1.00\n3.00\n5.00\n6.00\n▇▃▃▃▇\n\n\ntemperature\n0\n1\n0.50\n0.19\n0.02\n0.34\n0.50\n0.66\n1.00\n▂▇▇▇▁\n\n\napparent_temperature\n0\n1\n0.48\n0.17\n0.00\n0.33\n0.48\n0.62\n1.00\n▁▆▇▆▁\n\n\nhumidity\n0\n1\n0.63\n0.19\n0.00\n0.48\n0.63\n0.78\n1.00\n▁▃▇▇▆\n\n\nwindspeed\n0\n1\n0.19\n0.12\n0.00\n0.10\n0.19\n0.25\n0.85\n▇▆▂▁▁\n\n\ncount\n0\n1\n189.46\n181.39\n1.00\n40.00\n142.00\n281.00\n977.00\n▇▃▁▁▁\n\n\n\n\n\nThe original dataset does not contain a POSIXct column, but we can easily generate one based on the other variables available (note that as no information regarding minutes and seconds is available, we set them to :00:00):\n\nbike_sharing$date = as.POSIXct(paste0(bike_sharing$date, \" \", bike_sharing$hour, \":00:00\"),\n tz = \"GMT\", format = \"%Y-%m-%d %H:%M:%S\")" - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#baseline-random-forest", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#baseline-random-forest", - "title": "Feature Engineering of Date-Time Variables", - "section": "Baseline Random Forest", - "text": "Baseline Random Forest\nWe construct a new regression task and keep a holdout set.\n\ntask = as_task_regr(bike_sharing, target = \"count\")\n\nvalidation_set = sample(seq_len(task$nrow), size = 0.3 * task$nrow)\n\ntask$set_row_roles(validation_set, roles = \"holdout\")\n\nTo estimate the performance on unseen data, we will use a 3-fold cross-validation. Note that this involves validating on past data, which is usually bad practice but should suffice for this example:\n\ncv3 = rsmp(\"cv\", folds = 3)\n\nTo obtain reliable estimates on how well our model generalizes to the future, we would have to split our training and test sets according to the date variable.\nAs our baseline model, we use a random forest, ranger learner. For the baseline, we dropdate, our new POSIXct variable which we will only use later.\n\nlearner_ranger = lrn(\"regr.ranger\")\ntask_ranger = task$clone()\ntask_ranger$select(setdiff(task$feature_names, c(\"date\")))\n\nWe can then use resample() with 3-fold cross-validation:\n\nrr_ranger = resample(task_ranger, learner = learner_ranger, resampling = cv3)\n\nrr_ranger$score(msr(\"regr.mse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: bike_sharing regr.ranger cv 1\n2: bike_sharing regr.ranger cv 2\n3: bike_sharing regr.ranger cv 3\n prediction regr.mse\n1: 4412.082\n2: 4279.712\n3: 4849.678\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n iteration \n task_id \n learner_id \n resampling_id \n regr.mse \n \n \n\n \n 1 \n bike_sharing \n regr.ranger \n cv \n 4412.082 \n \n \n 2 \n bike_sharing \n regr.ranger \n cv \n 4279.712 \n \n \n 3 \n bike_sharing \n regr.ranger \n cv \n 4849.678 \n \n\n\n\n\nWe calculate the average RMSE.\n\nrr_ranger$aggregate()\n\nregr.mse \n4513.824 \n\n\nWe now want to improve our baseline model by using newly engineered features based on the date POSIXct column." - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#pipeopdatefeatures", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#pipeopdatefeatures", - "title": "Feature Engineering of Date-Time Variables", - "section": "PipeOpDateFeatures", - "text": "PipeOpDateFeatures\nTo engineer new features we use PipeOpDateFeatures. This pipeline automatically dispatches on POSIXct columns of the data and by default adds plenty of new date-time related features. Here, we want to add all except for minute and second, because this information is not available. As we additionally want to use cyclical versions of the features we set cyclic = TRUE:\n\npipeop_date = po(\"datefeatures\", cyclic = TRUE, minute = FALSE, second = FALSE)\n\nTraining this pipeline will result in simply adding the new features (and removing the original POSIXct feature(s) used for the feature engineering, see also the keep_date_var parameter). In our task, we can now drop the features, yr, mnth, hr, and weekday, because our pipeline will generate these anyways:\n\ntask_ex = task$clone()\ntask_ex$select(setdiff(task$feature_names,\n c(\"instant\", \"dteday\", \"yr\", \"mnth\", \"hr\", \"weekday\", \"casual\", \"registered\")))\n\npipeop_date$train(list(task_ex))\n\n$output\n (12166 x 32)\n* Target: count\n* Properties: -\n* Features (31):\n - dbl (23): apparent_temperature, date.day_of_month, date.day_of_month_cos, date.day_of_month_sin,\n date.day_of_week, date.day_of_week_cos, date.day_of_week_sin, date.day_of_year, date.day_of_year_cos,\n date.day_of_year_sin, date.hour, date.hour_cos, date.hour_sin, date.month, date.month_cos,\n date.month_sin, date.week_of_year, date.week_of_year_cos, date.week_of_year_sin, date.year, humidity,\n temperature, windspeed\n - lgl (3): date.is_day, holiday, working_day\n - int (3): hour, month, year\n - fct (2): season, weather\n\n\nNote that it may be useful to familiarize yourself with PipeOpRemoveConstants which can be used after the feature engineering to remove features that are constant. PipeOpDateFeatures does not do this step automatically.\nTo combine this feature engineering step with a random forest, ranger learner, we now construct a GraphLearner." - }, - { - "objectID": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#using-the-new-features-in-a-graphlearner", - "href": "gallery/2020-05-02-feature-engineering-of-date-time-variables/feature-engineering-of-date-time-variables.html#using-the-new-features-in-a-graphlearner", - "title": "Feature Engineering of Date-Time Variables", - "section": "Using the New Features in a GraphLearner", - "text": "Using the New Features in a GraphLearner\nWe create a GraphLearner consisting of the PipeOpDateFeatures pipeline and a ranger learner. This GraphLearner then behaves like any other Learner:\n\ngraph = po(\"datefeatures\", cyclic = TRUE, minute = FALSE, second = FALSE) %>>%\n lrn(\"regr.ranger\")\n\ngraph_learner = as_learner(graph)\n\nplot(graph, html = TRUE)\n\n\n\n\n\nUsing resample() with 3-fold cross-validation on the task yields:\n\ntask_graph_learner = task$clone()\ntask_graph_learner$select(setdiff(task$feature_names,\n c(\"instant\", \"dteday\", \"yr\", \"mnth\", \"hr\", \"weekday\", \"casual\", \"registered\")))\n\nrr_graph_learner = resample(task_graph_learner, learner = graph_learner, resampling = cv3)\n\nrr_graph_learner$score(msr(\"regr.mse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: bike_sharing datefeatures.regr.ranger cv 1\n2: bike_sharing datefeatures.regr.ranger cv 2\n3: bike_sharing datefeatures.regr.ranger cv 3\n prediction regr.mse\n1: 2528.759\n2: 2277.787\n3: 2252.469\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWe calculate the average RMSE.\n\nrr_graph_learner$aggregate()\n\nregr.mse \n2353.005 \n\n\nand therefore improved by almost 94%!\nFinally, we fit our GraphLearner on the complete training set and predict on the validation set:\n\ntask$select(setdiff(task$feature_names, c(\"year\", \"month\", \"hour\", \"weekday\")))\n\ngraph_learner$train(task)\n\nprediction = graph_learner$predict(task, row_ids = task$row_roles$validation)\n\nWhere we can obtain the RMSE on the held-out validation data.\n\nprediction$score(msr(\"regr.mse\"))\n\nregr.mse \n457.6439" - }, - { - "objectID": "gallery/2020-08-13-a-production-example-using-plumber-and-docker/a-production-example-using-plumber-and-docker.html", - "href": "gallery/2020-08-13-a-production-example-using-plumber-and-docker/a-production-example-using-plumber-and-docker.html", - "title": "A Production Example Using Plumber and Docker", - "section": "", - "text": "Production with R has come a long way. In this tutorial, we give a brief example on how to write a REST API and deploy it (relying on the mlr3 ecosystem for the actual training and predicting). Most of this tutorial was inspired by other excellent posts and vignettes:\n\nR can API and So Can You!\nUsing docker to deploy an R plumber API\nAzureContainer’s vignette\n\nAll files presented in this tutorial are available here.\n\nModeling Background\n\nlibrary(data.table)\nlibrary(mlr3)\nlibrary(mlr3pipelines)\n\nWe will use a subset of the boston_housing Task. Our goal is to predict the median value of owner-occupied homes in USD 1000’s (target medv), using the features crim, tax and town (just to have factor, integer, and numeric feature types):\n\ndata = tsk(\"boston_housing\")$data()\ndata = data[, c(\"medv\", \"crim\", \"tax\", \"town\")]\n\n\ntask = TaskRegr$new(\"boston\", backend = data, target = \"medv\")\n\nLet’s create a toy pipeline:\nRegarding modeling, we will keep it very simple and use the rpart learner. Missing numerical features (which could happen during prediction) will be imputed by their median via PipeOpImputeMedian, while missing factorial features will be imputed using a new level via PipeOpImputeOOR. As PipeOpImputeOOR will introduce a new level, \".MISSING\" to impute missing values, we also use PipeOpFixFactors:\n\ng = po(\"imputemedian\") %>>%\n po(\"imputeoor\") %>>%\n po(\"fixfactors\") %>>%\n lrn(\"regr.rpart\")\n\nWe wrap this Graph in a GraphLearner and can train on the Task:\n\ngl = GraphLearner$new(g)\ngl$train(task)\n\nWe can inspect the trained pipeline looking at:\n\ngl$model\n\nFurthermore, we can save the trained pipeline, i.e., as \"gl.rds\":\n\nsaveRDS(gl, \"gl.rds\")\n\nWe will also store some information regarding the features, i.e., the feature names, types and levels (you will later see, why we need to do this):\n\nfeature_info = list(\n feature_names = task$feature_names,\n feature_types = task$feature_types,\n levels = task$levels()\n)\nsaveRDS(feature_info, \"feature_info.rds\")\n\nPutting everything in a file, train_gl.R looks like the following, which we can then source before moving on:\n\n# train_gl.R\n\nlibrary(mlr3)\nlibrary(mlr3pipelines)\n\ndata = tsk(\"boston_housing\")$data()\ndata = data[, c(\"medv\", \"crim\", \"tax\", \"town\")]\ntask = TaskRegr$new(\"boston\", backend = data, target = \"medv\")\n\ng = po(\"imputemedian\") %>>%\n po(\"imputeoor\") %>>%\n po(\"fixfactors\") %>>%\n lrn(\"regr.rpart\")\n\ngl = GraphLearner$new(g)\n\ngl$train(task)\n\nsaveRDS(gl, \"gl.rds\")\n\nfeature_info = list(\n feature_names = task$feature_names,\n feature_types = task$feature_types,\n levels = task$levels()\n)\n\nsaveRDS(feature_info, \"feature_info.rds\")\n\nOur goal of our REST (representational state transfer) API (application programming interface) will be to predict the medv of a new observation, i.e., it should do something like the following:\n\nnewdata = data.table(crim = 3.14, tax = 691, town = \"Newton\")\ngl$predict_newdata(newdata)\n\n for 1 observations:\n row_ids truth response\n 1 NA 32.23288\n\n\nHowever, in our REST API, the newdata will be received at an endpoint that accepts a particular input. In the next section we will use plumber to set up our web service.\n\n\nUsing plumber to set up our REST API\nThe package plumber allows us to create a REST API by simply commenting existing R code. plumber makes use of these comments to define the web service. Running plumber::plumb on the commented R file then results in a runnable web service that other systems can interact with over a network.\nAs an endpoint for predicting the medv, we will use a POST request. This will allow us to enclose data in the body of the request message. More precisely, we assume that the data will be provided in the JSON format.\nWhen a POST request containing the data (in JSON format) is received our code must then:\n\nconvert the input (in JSON format) to a data.table with all feature columns matching their feature type\npredict the medv based on the input using our trained pipeline and provide an output that can be understood by the client\n\nTo make sure that all features match their feature type, we will later use the following function stored in the R file fix_feature_types.R:\n\n# fix_feature_types.R\n\nfix_feature_types = function(feature, feature_name, feature_info) {\n id = match(feature_name, feature_info$feature_names)\n feature_type = feature_info$feature_types$type[id]\n switch(feature_type,\n \"logical\" = as.logical(feature),\n \"integer\" = as.integer(feature),\n \"numeric\" = as.numeric(feature),\n \"character\" = as.character(feature),\n \"factor\" = factor(feature, levels = feature_info$levels[[feature_name]],\n ordered = FALSE),\n \"ordered\" = factor(feature, levels = feature_info$levels[[feature_name]],\n ordered = TRUE),\n \"POSIXct\" = as.POSIXct(feature)\n )\n}\n\nfix_feature_types() can later be applied to the newdata, and will make sure, that all incoming features are converted to their expected feature type as in the original Task we used for training our pipeline (and this is the reason, why we stored the information about the features earlier). Note that in our tutorial we only have factor, integer, and numeric features, but fix_feature_types() should also work for all other supported feature_types listed in mlr_reflections$task_feature_types. However, it may need some customization depending on your own production environment to make the conversions meaningful.\nThe following R file, predict_gl.R loads our trained pipepline and feature information and provides an endpoint for a POST request, \"/predict_medv\". The incoming data then is converted using jsonlite::fromJSON. We expect the incoming data to either be JSON objects in an array or nested JSON objects and therefore we bind the converted vectors row-wise to a data.table using data.table::rbindlist. We then convert all features to their expected feature_types (using the fix_feature_types() function as defined above) and can finally predict the medv using our trained pipeline. As no default serialization from R6 objects to JSON objects exists (yet), we wrap the Prediction in a data.table (of course we could also only return the numeric prediction values):\n\n# predict_gl.R\n\nlibrary(data.table)\nlibrary(jsonlite)\nlibrary(mlr3)\nlibrary(mlr3pipelines)\n\nsource(\"fix_feature_types.R\")\n\ngl = readRDS(\"gl.rds\")\n\nfeature_info = readRDS(\"feature_info.rds\")\n\n#* @post /predict_medv\nfunction(req) {\n # get the JSON string from the post body\n newdata = fromJSON(req$postBody, simplifyVector = FALSE)\n # expect either JSON objects in an array or nested JSON objects\n newdata = rbindlist(newdata, use.names = TRUE)\n # convert all features in place to their expected feature_type\n newdata[, colnames(newdata) := mlr3misc::pmap(\n list(.SD, colnames(newdata)),\n fix_feature_types,\n feature_info = feature_info)]\n # predict and return as a data.table\n as.data.table(gl$predict_newdata(newdata))\n # or only the numeric values\n # gl$predict_newdata(newdata)$response\n}\n\nNote that the only difference to a regular R file is the comment\n\n#* @post /predict_medv`\n\ntelling plumber to construct the endpoint \"/predict_medv\" for a POST request.\nWe can then run plumber::plumb. The following code sets up the web service locally on your personal machine at port 1030 (we use such a high number because some systems require administrator rights to allow processes to listen to lower ports):\n\nlibrary(plumber)\nr = plumb(\"predict_gl.R\")\nr$run(port = 1030, host = \"0.0.0.0\")\n\nCongratulations, your first REST API is running on your local machine. We can test it by providing some data, using curl via the command line:\n\ncurl --data '[{\"crim\":3.14, \"tax\":691, \"town\":\"Newton\"}]' \"http://127.0.0.1:1030/predict_medv\"\n\nThis should return the predicted medv:\n\n[{\"row_id\":1,\"response\":\"32.2329\"}]\n\nAlternatively, we can also use the httr::POST function within R:\n\nnewdata = '[{\"crim\":3.14, \"tax\":691, \"town\":\"Newton\"}]'\nresp = httr::POST(url = \"http://127.0.0.1:1030/predict_medv\",\n body = newdata, encode = \"json\")\nhttr::content(resp)\n\nWe can further play around a bit more and provide more than a single new observation and also check whether our feature type conversion and missing value imputation works:\n\nnewdata = '[\n {\"crim\":3.14, \"tax\":691, \"town\":\"Newton\"},\n {\"crim\":\"not_a_number\", \"tax\":3.14, \"town\":\"Munich\"},\n {\"tax\":\"not_a_number\", \"town\":31, \"crim\":99}\n]'\nresp = httr::POST(url = \"http://127.0.0.1:1030/predict_medv\",\n body = newdata, encode = \"json\")\nhttr::content(resp)\n\nNote that you can also use jsonlite::toJSON to convert a data.frame to JSON data for your toy examples here.\nIn the following final section we want to use Docker to run a virtual machine as a container (an instance of a snapshot of a machine at a moment in time).\n\n\nUsing Docker to Deploy our REST API\nA Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application. Suppose we want to run our REST API on an Amazon Web Service or Microsoft’s Azure Cloud. Then we can use a Docker container to easily set up our web service without going through the hassle of configuring manually our hosting instance.\nWe are going to need two things: An image and a container. An image defines the OS and software while the container is the actual running instance of the image. To build a Docker image we have to specify a Dockerfile. Note that it is sensible to set up the whole project in its own directory, e.g., ~/mlr3_api.\nEvery Dockerfile starts with a FROM statement describing the image we are building our image from. In our case we want an R based image that ideally already has plumber and its dependencies installed. Luckily, the trestletech/plumber image exists:\n\nFROM trestletech/plumber\n\nWe then install the R packages needed to set up our REST API (note that we can skip jsonlite, because plumber already depends on it):\n\nRUN R -e 'install.packages(c(\"data.table\", \"mlr3\", \"mlr3pipelines\"))'\n\nNext, we copy our trained pipeline (gl.rds), our stored feature information (feature_info.rds), our R file to convert features, (fix_feature_types.R) and our R file to predict (predict_gl.R) to a new directory /data and set this as the working directory:\n\nRUN mkdir /data\nCOPY gl.rds /data\nCOPY feature_info.rds /data\nCOPY fix_feature_types.R /data\nCOPY predict_gl.R /data\nWORKDIR /data\n\nFinally, we listen on port 1030 and start the server (this is analogously done as manually calling plumber::plumb on the predict_gl.R file and running it):\n\nEXPOSE 1030\nENTRYPOINT [\"R\", \"-e\", \\\n \"r = plumber::plumb('/data/predict_gl.R'); r$run(port = 1030, host = '0.0.0.0')\"]\n\nThe complete Dockerfile looks like the following:\n\nFROM trestletech/plumber\n\nRUN R -e 'install.packages(c(\"data.table\", \"mlr3\", \"mlr3misc\", \"mlr3pipelines\"))'\n\nRUN mkdir /data\nCOPY gl.rds /data\nCOPY feature_info.rds /data\nCOPY fix_feature_types.R /data\nCOPY predict_gl.R /data\nWORKDIR /data\n\nEXPOSE 1030\nENTRYPOINT [\"R\", \"-e\", \\\n \"r = plumber::plumb('/data/predict_gl.R'); r$run(port = 1030, host = '0.0.0.0')\"]\n\nTo build the image we open a terminal in the mlr3_api directory and run:\n\ndocker build -t mlr3-plumber-demo .\n\nThis may take quite some time.\nTo finally run the container, simply use:\n\ndocker run --rm -p 1030:1030 mlr3-plumber-demo\n\nYou can then proceed to provide some data via curl or httr::POST (to the same local address, because the Docker container is still running on your local machine).\nTo stop all running containers use:\n\ndocker stop $(docker ps -a -q)\n\nFinally, you can proceed to deploy your container to an Amazon Web Service or an Azure Cloud. For the latter, the package AzureContainers is especially helpful. If you do plan to do this note that the plumber service above is exposed over HTTP, and there is no authentication layer making it insecure. You may think about adding a layer of authentification and restricting the service to HTTPS." - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "", - "text": "This is the second post of the titanic use case series. You can find the first use case here.\nIn this section we will focus on more advanced usage of mlr3pipelines . Specifically, this section illustrates the different options when it comes to data imputation and feature engineering. Furthermore, the section shows how to benchmark, feature engineer and compare our results.\nWe load the mlr3verse package which pulls in the most important packages for this example. The mlr3learners package loads additional learners. The data is part of the mlr3data package.\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\nAs in the basics chapter, we use the titanic data set. To recap we have undertaken the following steps:" - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#imputation", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#imputation", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "Imputation", - "text": "Imputation\nA very simple way to do this to just impute a constant value for each feature. We could i.e. impute every character or factor column with missing and every numeric column with -999. And depending on the model, this might actually be fine. This approach has a few drawbacks though:\n\n-999 could be a real value in the data.\nimputing -999 skews the distribution of the data, which might result in bad models.\n\nAs a result, instead of imputing a constant value, we will do two things: * Draw samples from each numeric features’ histogram using PipeOpImputeHist * Add an additional column for each variable that indicates whether a value was missing or not. If the information that a value was missing is important, this column contains this information.\nThis imputation scheme is called ‘imputation with constants’ and is already implemented in mlr3pipelines . It can be done using PipeOpImputeConstant.\nRemember that we are trying to optimize our predictive power by using a random forest model (mlr_learners_classif.ranger). Now, random forest models do not naturally handle missing values which is the reason why we need imputation. Before imputation, our data looks as follows:\n\ntask$missings()\n\nsurvived age embarked fare parch pclass sex sib_sp \n 0 177 2 0 0 0 0 0 \n\n\nLet’s first deal with the categorical variables:\n\npo_newlvl = po(\"imputeoor\")\ntask_newlvl = po_newlvl$train(list(task))[[1]]\n\nNote that we use the PipeOp in an unusual way, which is why the syntax does not look very clean. We’ll learn how to use a full graph below.\nFirst, let’s look at the result:\n\ntask_newlvl$missings()\n\nsurvived fare parch pclass sex sib_sp age embarked \n 0 0 0 0 0 0 0 0 \n\n\nCool! embarked does not have missing values anymore. Note that PipeOpImputeOOR by default affects character, factor and ordered columns.\nFor the numeric features we want to do two things, impute values and add an indicator column. In order to do this, we need a more complicated structure, a Graph.\nOur po_indicator creates the indicator column. We tell it to only do this for numeric and integer columns via its param_vals, and additionally tell it to create a numeric column (0 = “not missing”, 1 = “missing”).\n\npo_indicator = po(\"missind\",\n affect_columns = selector_type(c(\"numeric\", \"integer\")), type = \"numeric\")\n\nNow we can simultaneously impute features from the histogram and create indicator columns. This can be achieved using the gunion function, which puts two operations in parallel:\n\ngraph = gunion(list(po_indicator, po(\"imputehist\")))\ngraph = graph %>>% po(\"featureunion\")\n\nAfterwards, we cbind the resulting data using po(\"featureunion\"), connecting the different operations using our graph connector: %>>%. We can now also connect the newlvl imputation:\n\ngraph = graph %>>% po(\"imputeoor\")\n\nand see what happens when we now train the whole Graph:\n\ntask_imputed = graph$clone()$train(task)[[1]]\ntask_imputed$missings()\n\n survived missing_age pclass sex fare parch sib_sp age embarked \n 0 0 0 0 0 0 0 0 0 \n\n\nAwesome, now we do not have any missing values!\n\nautoplot(task_imputed)\n\n\n\n\n\n\n\n\nWe could now use task_imputed for resampling and see whether a ranger model does better. But this is dangerous! If we preprocess all training data at once, data could leak through the different cross-validation folds. In order to do this properly, we have to process the training data in every fold separately. Luckily, this is automatically handled in our Graph, if we use it through a GraphLearner.\nWe can simply append a ranger learner to the Graph and create a GraphLearner from this.\n\ngraph_learner = as_learner(graph$clone() %>>%\n po(\"imputesample\") %>>%\n po(\"fixfactors\") %>>%\n po(learner))\n\nWe needed to use the following commands for the Graph: * PipeOpFixFactors: Removes empty factor levels and removes factor levels that do not exist during training. * PipeOpImputeSample: In some cases, if missing factor levels do not occur during training but only while predicting, PipeOpImputeOOR does not create a new level. For those, we sample a random value.\n\nrr = resample(task, graph_learner, resampling, store_models = TRUE)\nrr$aggregate(msr(\"classif.acc\"))\n\nclassif.acc \n 0.7934905 \n\n\nSo our model has not improved heavily, currently it has an accuracy of 0.79." - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#feature-engineering", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#feature-engineering", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "Feature Engineering", - "text": "Feature Engineering\nWe will do this using PipeOpMutate in order to showcase the power of mlr3pipelines . Additionally, we will make use of the character columns. Hence, we will re-select them:\n\ntask$col_roles$feature = c(task$feature_names, c(\"cabin\", \"name\", \"ticket\"))\n\n\nlibrary(\"stringi\")\npo_ftextract = po(\"mutate\", mutation = list(\n fare_per_person = ~ fare / (parch + sib_sp + 1),\n deck = ~ factor(stri_sub(cabin, 1, 1)),\n title = ~ factor(stri_match(name, regex = \", (.*)\\\\.\")[, 2]),\n surname = ~ factor(stri_match(name, regex = \"(.*),\")[, 2]),\n ticket_prefix = ~ factor(stri_replace_all_fixed(stri_trim(stri_match(ticket, regex = \"(.*) \")[, 2]), \".\", \"\"))\n))\n\nQuickly checking what happens:\n\ntask_eng = po_ftextract$clone()$train(list(task))[[1]]\ntask_eng$data()\n\n survived age embarked fare parch pclass sex sib_sp cabin name\n 1: no 22 S 7.2500 0 3 male 1 Braund, Mr. Owen Harris\n 2: yes 38 C 71.2833 0 1 female 1 C85 Cumings, Mrs. John Bradley (Florence Briggs Thayer)\n 3: yes 26 S 7.9250 0 3 female 0 Heikkinen, Miss. Laina\n 4: yes 35 S 53.1000 0 1 female 1 C123 Futrelle, Mrs. Jacques Heath (Lily May Peel)\n 5: no 35 S 8.0500 0 3 male 0 Allen, Mr. William Henry\n --- \n887: no 27 S 13.0000 0 2 male 0 Montvila, Rev. Juozas\n888: yes 19 S 30.0000 0 1 female 0 B42 Graham, Miss. Margaret Edith\n889: no NA S 23.4500 2 3 female 1 Johnston, Miss. Catherine Helen \"Carrie\"\n890: yes 26 C 30.0000 0 1 male 0 C148 Behr, Mr. Karl Howell\n891: no 32 Q 7.7500 0 3 male 0 Dooley, Mr. Patrick\n ticket fare_per_person deck title surname ticket_prefix\n 1: A/5 21171 3.62500 Mr Braund A/5\n 2: PC 17599 35.64165 C Mrs Cumings PC\n 3: STON/O2. 3101282 7.92500 Miss Heikkinen STON/O2\n 4: 113803 26.55000 C Mrs Futrelle \n 5: 373450 8.05000 Mr Allen \n --- \n887: 211536 13.00000 Rev Montvila \n888: 112053 30.00000 B Miss Graham \n889: W./C. 6607 5.86250 Miss Johnston W/C\n890: 111369 30.00000 C Mr Behr \n891: 370376 7.75000 Mr Dooley \n\n\n\nautoplot(task_eng$clone()$select(c(\"sex\", \"age\")), type = \"pairs\")\n\nRegistered S3 method overwritten by 'GGally':\n method from \n +.gg ggplot2\n\n\nWarning: Removed 177 rows containing non-finite values (stat_boxplot).\n\n\n`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.\n\n\nWarning: Removed 177 rows containing non-finite values (stat_bin).\n\n\nWarning: Removed 177 rows containing non-finite values (stat_density).\n\n\nWarning: Removed 177 rows containing non-finite values (stat_boxplot).\n\n\n`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.\n\n\nWarning: Removed 177 rows containing non-finite values (stat_bin).\n\n\n\n\n\n\n\n\n\nNow we can put everything together again, we concatenate our new PipeOp with the Graph created above and use PipeOpSelect in order to de-select the character features we used for feature extraction. Additionally, we collapse the ‘surname’, so only surnames that make up more than 0.6 % of the data are kept.\nIn summary, we do the following:\n\nmutate: The po_ftextract we defined above extracts additional features from the data.\ncollapsefactors: Removes factor levels that make up less then 3 % of the data.\nselect: Drops character columns.\ngunion: Puts two PipeOps in parallel.\n\nmissind: po_indicator adds a column for each numeric with the info whether the value is NA or not.\nimputehist: Imputes numeric and integer columns by sampling from the histogram.\n\nfeatureunion: Cbind’s parallel data streams.\nimputeoor: Imputes factor and ordered columns.\nfixfactors: Removes empty factor levels and removes factor levels that do not exist during training.\nimputesample: In some cases, if missing factor levels do not occur during training but only while predicting, imputeoor does not create a new level. For those, we sample a random value.\nLearner: Appends a learner to the Graph.\n\nThe full graph we created is the following:\n\nlearner = lrn(\"classif.ranger\", num.trees = 500, min.node.size = 4)\n\n\ngraph_final = po_ftextract %>>%\n po(\"collapsefactors\", param_vals = list(no_collapse_above_prevalence = 0.03)) %>>%\n po(\"select\", param_vals = list(selector = selector_invert(selector_type(\"character\")))) %>>%\n gunion(list(po_indicator, po(\"imputehist\"))) %>>%\n po(\"featureunion\") %>>%\n po(\"imputeoor\") %>>%\n po(\"fixfactors\") %>>%\n po(\"imputesample\") %>>%\n po(learner)" - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#evaluation", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#evaluation", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "Evaluation", - "text": "Evaluation\nLet us see if things have improved:\n\ngraph_learner = as_learner(graph_final)\n\nrr = resample(task, graph_learner, resampling, store_models = TRUE)\n\nrr$aggregate(msr(\"classif.acc\"))\n\nclassif.acc \n 0.8226712 \n\n\nWe have improved even more!" - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#benchmarking", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#benchmarking", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "Benchmarking", - "text": "Benchmarking\nTo undertake benchmarking, we need to set up a benchmarking design. The first step is creating a list with the learners we used, namely the learners form the first and second part of this use case.\n\nlearners = list(\n lrn(\"classif.rpart\", predict_type = \"prob\"),\n lrn(\"classif.ranger\", predict_type = \"prob\")\n)\n\nNow we can define our benchmark design. This is done to ensure exhaustive and consistent resampling for all learners. This step is needed to execute over the same train/test split for each task.\n\nbm_design = benchmark_grid(task_imputed, learners, rsmp(\"cv\", folds = 10))\nbmr = benchmark(bm_design, store_models = TRUE)\nprint(bmr)\n\n of 20 rows with 2 resampling runs\n nr task_id learner_id resampling_id iters warnings errors\n 1 titanic classif.rpart cv 10 0 0\n 2 titanic classif.ranger cv 10 0 0\n\n\nSo, where do we go from here? We could for instance use a boxplot:\n\nautoplot(bmr)\n\n\n\n\n\n\n\n\nFurther we are able to compare sensitivity and specificity. Here we need to ensure that the benchmark results only contain a single Task:\n\nautoplot(bmr$clone()$filter(task_id = \"titanic\"), type = \"roc\")\n\n\n\n\n\n\n\n\nMoreover, one can compare the precision-recall:\n\n# Precision vs Recall\nggplot2::autoplot(bmr, type = \"prc\")\n\n\n\n\n\n\n\n\nAs one can see, there are various options when it comes to benchmarking and visualizing. You could have a look at some other use cases in our gallery for inspiration." - }, - { - "objectID": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#future", - "href": "gallery/2020-04-27-mlr3pipelines-Imputation-titanic/2020-04-27-mlr3pipelines-titanic-advanced.html#future", - "title": "A Pipeline for the Titanic Data Set - Advanced", - "section": "Future", - "text": "Future\nIn this case we have examined a number of different features, but there are many more things to explore! We could extract even more information from the different features and see what happens. But now you are left to yourself! There are many kaggle kernels that treat the Titanic Dataset available. This can be a great starter to find even better models." - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "", - "text": "This use case shows how to easily work with datasets available via OpenML into an mlr3 workflow.\nThe following operations are illustrated:" - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#prerequisites", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#prerequisites", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Prerequisites", - "text": "Prerequisites\n\n# various functions of the mlr3 ecosystem\nlibrary(\"mlr3verse\")\n# about a dozen reasonable learners\nlibrary(\"mlr3learners\")\n# retrieving the data\nlibrary(\"OpenML\")\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#retrieving-the-data-from-openml", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#retrieving-the-data-from-openml", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Retrieving the data from OpenML", - "text": "Retrieving the data from OpenML\nWe can use the OpenML package to retrieve data (and more) straight away. OpenML is is an inclusive movement to build an open, organized, online ecosystem for machine learning. Typically, you can retrieve the data with an data.id. The id can be found on OpenML. We choose the data set 41021. The related web page can be accessed here. This data set was uploaded by Joaquin Vanschoren.\n\noml_data = getOMLDataSet(data.id = 41021)\n\nDownloading from 'http://www.openml.org/api/v1/data/41021' to '/var/folders/nr/x23mhfm55616f3w8xd0lwmdh0000gn/T//RtmpxRsNvX/cache/datasets/41021/description.xml'.\n\n\nDownloading from 'https://api.openml.org/data/v1/download/18626236/Moneyball.arff' to '/var/folders/nr/x23mhfm55616f3w8xd0lwmdh0000gn/T//RtmpxRsNvX/cache/datasets/41021/dataset.arff'\n\n\nLoading required package: readr\n\n\nThe description indicates that the data set is associated with baseball or more precisely the story of Moneyball.\n\nhead(as.data.table(oml_data))\n\n\n\n\n\n\n\n\nHowever, the description within the OpenML object is not very detailed. The previously referenced page however states the following:\nIn the early 2000s, Billy Beane and Paul DePodesta worked for the Oakland Athletics. During their work there, they disrupted the game of baseball. They didn’t do it using a bat or glove, and they certainly didn’t do it by throwing money at the issue; in fact, money was the issue. They didn’t have enough of it, but they were still expected to keep up with teams that had more substantial endorsements. This is where Statistics came riding down the hillside on a white horse to save the day. This data set contains some of the information that was available to Beane and DePodesta in the early 2000s, and it can be used to better understand their methods.\nThis data set contains a set of variables that Beane and DePodesta emphasized in their work. They determined that statistics like on-base percentage (obp) and slugging percentage (slg) were very important when it came to scoring runs, however, they were largely undervalued by most scouts at the time. This translated to a gold mine for Beane and DePodesta. Since these players weren’t being looked at by other teams, they could recruit these players on a small budget. The variables are as follows:\n\nteam\nleague\nyear\nruns scored (rs)\nruns allowed (ra)\nwins (w)\non-base percentage (obp)\nslugging percentage (slg)\nbatting average (ba)\nplayoffs (binary)\nrankseason\nrankplayoffs\ngames played (g)\nopponent on-base percentage (oobp)\nopponent slugging percentage (oslg)\n\nWhile Beane and DePodesta defined most of these statistics and measures for individual players, this data set is on the team level.\nThese statistics seem very informative if you are into baseball. If baseball of rather obscure to you, simply take these features as given or give this article a quick read.\nFinally, note that the moneyball dataset is also included in the mlr3data package where you can get the preprocessed (integers properly encoded as such, etc.) data via:\n\ndata(\"moneyball\", package = \"mlr3data\")\nskimr::skim(moneyball)\n\n\nData summary\n\n\nName\nmoneyball\n\n\nNumber of rows\n1232\n\n\nNumber of columns\n15\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n6\n\n\nnumeric\n9\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\nteam\n0\n1.0\nFALSE\n39\nBAL: 47, BOS: 47, CHC: 47, CHW: 47\n\n\nleague\n0\n1.0\nFALSE\n2\nAL: 616, NL: 616\n\n\nplayoffs\n0\n1.0\nFALSE\n2\n0: 988, 1: 244\n\n\nrankseason\n988\n0.2\nFALSE\n8\n2: 53, 1: 52, 3: 44, 4: 44\n\n\nrankplayoffs\n988\n0.2\nFALSE\n5\n3: 80, 4: 68, 1: 47, 2: 47\n\n\ng\n0\n1.0\nFALSE\n8\n162: 954, 161: 139, 163: 93, 160: 23\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nyear\n0\n1.00\n1988.96\n14.82\n1962.00\n1976.75\n1989.00\n2002.00\n2012.00\n▆▆▇▆▇\n\n\nrs\n0\n1.00\n715.08\n91.53\n463.00\n652.00\n711.00\n775.00\n1009.00\n▁▆▇▃▁\n\n\nra\n0\n1.00\n715.08\n93.08\n472.00\n649.75\n709.00\n774.25\n1103.00\n▂▇▆▂▁\n\n\nw\n0\n1.00\n80.90\n11.46\n40.00\n73.00\n81.00\n89.00\n116.00\n▁▃▇▆▁\n\n\nobp\n0\n1.00\n0.33\n0.02\n0.28\n0.32\n0.33\n0.34\n0.37\n▁▃▇▅▁\n\n\nslg\n0\n1.00\n0.40\n0.03\n0.30\n0.38\n0.40\n0.42\n0.49\n▁▅▇▅▁\n\n\nba\n0\n1.00\n0.26\n0.01\n0.21\n0.25\n0.26\n0.27\n0.29\n▁▂▇▆▂\n\n\noobp\n812\n0.34\n0.33\n0.02\n0.29\n0.32\n0.33\n0.34\n0.38\n▂▇▇▃▁\n\n\noslg\n812\n0.34\n0.42\n0.03\n0.35\n0.40\n0.42\n0.44\n0.50\n▁▆▇▅▁\n\n\n\n\n\nThe summary shows how this data we are dealing with looks like: Some data is missing, however, this has structural reasons. There are 39 teams with each maximally 47 years (1962 - 2012). For 988 cases the information on rankseason and rankplayoffs is missing. This is since these simply did not reach the playoffs and hence have no reported rank.\n\nsummary(moneyball[moneyball$playoffs == 0, c(\"rankseason\", \"rankplayoffs\")])\n\n rankseason rankplayoffs\n 1 : 0 1 : 0 \n 2 : 0 2 : 0 \n 3 : 0 3 : 0 \n 4 : 0 4 : 0 \n 5 : 0 5 : 0 \n (Other): 0 NA's:988 \n NA's :988 \n\n\nOn the other hand, oobp and oslg have \\(812\\) missing values. It seems as if these measures were not available before \\(1998\\).\n\nlibrary(ggplot2)\nlibrary(naniar)\n\nggplot(moneyball, aes(x = year, y = oobp)) +\n geom_miss_point()\n\n\n\n\n\n\n\n\nWe seem to have a missing data problem. Typically, in this case, we have three options: They are:\n\nComplete case analysis: Exclude all observation with missing values.\nComplete feature analysis: Exclude all features with missing values.\nMissing value imputation: Use a model to “guess” the missing values (based on the underlying distribution of the data.\n\nUsually, missing value imputation is preferred over the first two. However, in machine learning, one can try out all options and see which performs best for the underlying problem. For now, we limit ourselves to a rather simple imputation technique, imputation by randomly sampling from the univariate distribution. Note that this does not take the multivariate distribution into account properly and that there are more elaborate approaches. We only aim to impute oobp and oslg. For the other missing (categorical) features, we simply add a new level which indicates that information is missing (i.e. all missing values belong to).\nIt is important to note that in this case here the vast majority of information on the features is missing. In this case, imputation is performed to not throw away the existing information of the features.\nmlr3 has some solutions for that within the mlr3pipelines package. We start with an easy PipeOp which only performs numeric imputation.\n\nimp_num = po(\"imputehist\", affect_columns = selector_type(c(\"integer\", \"numeric\")))\n\nNext, we append the second imputation job for factors.\n\nimp_fct = po(\"imputeoor\", affect_columns = selector_type(\"factor\"))\ngraph = imp_num %>>% imp_fct\ngraph$plot(html = TRUE)" - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#creating-tasks-and-learners", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#creating-tasks-and-learners", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Creating tasks and learners", - "text": "Creating tasks and learners\nThe fact that there is missing data does not affect the task definition. The task determines what is the problem to be solved by machine learning. We want to explain the runs scored (rs). rs is an important measure as a run is equivalent to a ‘point’ scored in other sports. Naturally, the aim of a coach should be to maximise runs scored and minimise runs allowed. As runs scored and runs allowed are both legitimate targets we ignore the runs allowed here. The task is defined by:\n\n# creates a `mlr3` task from scratch, from a data.frame\n# 'target' names the column in the dataset we want to learn to predict\ntask = as_task_regr(moneyball, target = \"rs\")\ntask$missings()\n\n\n\n\n\n\n\n\nThe $missings() method indicates what we already knew: our missing values. Missing values are not always a problem. Some learners can deal with them pretty well. However, we want to use a random forest for our task.\n\n# creates a learner\ntest_learner = lrn(\"regr.ranger\")\n\n# displays the properties\ntest_learner$properties\n\n[1] \"hotstart_backward\" \"importance\" \"oob_error\" \"weights\" \n\n\nTypically, in mlr3 the $properties field would tell us whether missing values are a problem to this learner or not. As it is not listed here, the random forest cannot deal with missing values.\nAs we aim to use imputation beforehand, we incorporate it into the learner. Our selected learner is going to be a random forest from the ranger package.\nOne can allow the embedding of the preprocessing (imputation) into a learner by creating a PipeOpLearner. This special Learner can be put into a graph together with the imputer.\n\n# convert learner to pipeop learner and set hyperparameter\npipeop_learner = po(lrn(\"regr.ranger\"), num.trees = 1000, importance = \"permutation\")\n\n# add pipeop learner to graph and create graph learner\ngraph_learner = as_learner(graph %>>% pipeop_learner)\n\nThe final graph looks like the following:\n\ngraph_learner$graph$plot(html = TRUE)" - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#train-and-predict", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#train-and-predict", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Train and predict", - "text": "Train and predict\nTo get a feeling of how our model performs we simply train the Learner on a subset of the data and predict the hold-out data.\n\n# defines the training and testing data; 95% is used for training\ntrain_set = sample(task$nrow, 0.95 * task$nrow)\ntest_set = setdiff(seq_len(task$nrow), train_set)\n\n# train learner on subset of task\ngraph_learner$train(task, row_ids = train_set)\n\n# predict using held out observations\nprediction = graph_learner$predict(task, row_ids = test_set)\n\nhead(as.data.table(prediction))\n\n\n\n\n\n\n\n\nViewing the predicted values it seems like the model predicts reasonable values that are fairly close to the truth." - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#evaluation-resampling", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#evaluation-resampling", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Evaluation & Resampling", - "text": "Evaluation & Resampling\nWhile the prediction indicated that the model is doing what it is supposed to, we want to have a more systematic understanding of the model performance. That means we want to know by how much our model is away from the truth on average. Cross-validation investigates this question. In mlr3 10-fold cross-validation is constructed as follows:\n\ncv10 = rsmp(\"cv\", folds = 10)\nrr = resample(task, graph_learner, cv10)\n\nWe choose some of the performance measures provided by:\n\nas.data.table(mlr_measures)\n\n\n\n\n\n\n\n\nWe choose the mean absolute error and the mean squared error.\n\nrr$score(msrs(c(\"regr.mae\", \"regr.mse\")))\n\n\n\n\n\n\n\n\nWe can also compute now by how much our model was on average wrong when predicting the runs scored.\n\nrr$aggregate(msr(\"regr.mae\"))\n\nregr.mae \n 19.3932 \n\n\nThat seems not too bad. Considering that on average approximately 715 runs per team per season have been scored.\n\nmean(moneyball$rs)\n\n[1] 715.082" - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#performance-boost-of-imputation", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#performance-boost-of-imputation", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Performance boost of imputation", - "text": "Performance boost of imputation\nTo assess if imputation was beneficial, we can compare our current learner with a learner which ignores the missing features. Normally, one would set up a benchmark for this. However, we want to keep things short in this use case. Thus, we only set up the alternative learner (with identical hyperparameters) and compare the 10-fold cross-validated mean absolute error.\nAs we are mostly interested in the numeric imputation we leave the remaining graph as it is.\n\nimpute_oor = po(\"imputeoor\", affect_columns = selector_type(\"factor\"))\n\nSubsequently, we create a pipeline with PipeOpSelect.\n\nfeature_names = colnames(moneyball)[!sapply(moneyball, anyNA)]\nfeature_names = c(feature_names[feature_names %in% task$feature_names],\n \"rankseason\", \"rankplayoffs\")\nselect_na = po(\"select\", selector = selector_name(feature_names))\n\ngraph_2 = impute_oor %>>% select_na\ngraph_2$plot(html = TRUE)\n\n\n\n\n\nNow we complete the learner and apply resampling as before.\n\ngraph_learner_2 = as_learner(graph_2 %>>% pipeop_learner)\nrr_2 = resample(task, graph_learner_2, cv10)\nrr_2$aggregate(msr(\"regr.mae\"))\n\nregr.mae \n19.02211 \n\n\nSurprisingly, the performance seems to be approximately the same. That means that the imputed features seem not very helpful. We can use the variable.importance of the random forest.\n\nsort(graph_learner$model$regr.ranger$model$variable.importance, decreasing = TRUE)\n\n\n\n\n\n\n\n\nWe see that according to this the left out oobp and oslg seem to have solely rudimentary explanatory power. This may be because there were simply too many instances or because the features are themselves not very powerful." - }, - { - "objectID": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#conclusion", - "href": "gallery/2020-05-04-moneyball/2020-05-04-moneyball.html#conclusion", - "title": "mlr3 and OpenML - Moneyball Use Case", - "section": "Conclusion", - "text": "Conclusion\nSo, to sum up, what we have learned: We can access very cool data straight away with the OpenML package. (We are working on a better direct implementation into mlr3 at the moment.) We can work with missing data very well in mlr3. Nevertheless, we discovered that sometimes imputation does not lead to the intended goals. We also learned how to use some PipeOps from the mlr3pipelines package.\nBut most importantly, we found a way to predict the runs scored of MLB teams.\nIf you want to know more, read the mlr3book and the documentation of the mentioned packages." - }, - { - "objectID": "gallery/2021-03-11-practical-tuning-series-build-an-automated-machine-learning-system/practical-tuning-series-build-an-automated-machine-learning-system.html", - "href": "gallery/2021-03-11-practical-tuning-series-build-an-automated-machine-learning-system/practical-tuning-series-build-an-automated-machine-learning-system.html", - "title": "Practical Tuning Series - Build an Automated Machine Learning System", - "section": "", - "text": "Scope\nThis is the third part of the practical tuning series. The other parts can be found here:\n\nPart I - Tune a Support Vector Machine\nPart II - Tune a Preprocessing Pipeline\nPart IV - Tuning and Parallel Processing\n\nIn this post, we implement a simple automated machine learning (AutoML) system which includes preprocessing, a switch between multiple learners and hyperparameter tuning. For this, we build a pipeline with the mlr3pipelines extension package. Additionally, we use nested resampling to get an unbiased performance estimate of our AutoML system.\n\n\nPrerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented. The lgr package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning ).\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nIn this example, we use the Pima Indians Diabetes data set which is used to to predict whether or not a patient has diabetes. The patients are characterized by 8 numeric features and some have missing values.\n\ntask = tsk(\"pima\")\n\n\n\nBranching\nWe use three popular machine learning algorithms: k-nearest-neighbors, support vector machines and random forests.\n\nlearners = list(\n lrn(\"classif.kknn\", id = \"kknn\"),\n lrn(\"classif.svm\", id = \"svm\", type = \"C-classification\"),\n lrn(\"classif.ranger\", id = \"ranger\")\n)\n\nThe PipeOpBranch allows us to specify multiple alternatives paths. In this graph, the paths lead to the different learner models. The selection hyperparameter controls which path is executed i.e., which learner is used to fit a model. It is important to use the PipeOpBranch after the branching so that the outputs are merged into one result object. We visualize the graph with branching below.\n\ngraph =\n po(\"branch\", options = c(\"kknn\", \"svm\", \"ranger\")) %>>%\n gunion(lapply(learners, po)) %>>%\n po(\"unbranch\")\ngraph$plot(html = TRUE)\n\n\n\n\n\nAlternatively, we can use the ppl()-shortcut to load a predefined graph from the mlr_graphs dictionary. For this, the learner list must be named.\n\nlearners = list(\n kknn = lrn(\"classif.kknn\", id = \"kknn\"),\n svm = lrn(\"classif.svm\", id = \"svm\", type = \"C-classification\"),\n ranger = lrn(\"classif.ranger\", id = \"ranger\")\n)\n\ngraph = ppl(\"branch\", lapply(learners, po))\n\n\n\nPreprocessing\nThe task has missing data in five columns.\n\nround(task$missings() / task$nrow, 2)\n\ndiabetes age glucose insulin mass pedigree pregnant pressure triceps \n 0.00 0.00 0.01 0.49 0.01 0.00 0.00 0.05 0.30 \n\n\nThe pipeline \"robustify\" function creates a preprocessing pipeline based on our task. The resulting pipeline imputes missing values with PipeOpImputeHist and creates a dummy column (PipeOpMissInd) which indicates the imputed missing values. Internally, this creates two paths and the results are combined with PipeOpFeatureUnion. In contrast to PipeOpBranch, both paths are executed. Additionally, \"robustify\" adds PipeOpEncode to encode factor columns and PipeOpRemoveConstants to remove features with a constant value.\n\ngraph = ppl(\"robustify\", task = task, factors_to_numeric = TRUE) %>>%\n graph\nplot(graph, html = TRUE)\n\n\n\n\n\nWe could also create the preprocessing pipeline manually.\n\ngunion(list(po(\"imputehist\"),\n po(\"missind\", affect_columns = selector_type(c(\"numeric\", \"integer\"))))) %>>%\n po(\"featureunion\") %>>%\n po(\"encode\") %>>%\n po(\"removeconstants\")\n\nGraph with 5 PipeOps:\n ID State sccssors prdcssors\n imputehist <> featureunion \n missind <> featureunion \n featureunion <> encode imputehist,missind\n encode <> removeconstants featureunion\n removeconstants <> encode\n\n\n\n\nGraph Learner\nWe use as_learner() to create a GraphLearner which encapsulates the pipeline and can be used like a learner.\n\ngraph_learner = as_learner(graph)\n\nThe parameter set of the graph learner includes all hyperparameters from all contained learners. The hyperparameter ids are prefixed with the corresponding learner ids. The hyperparameter branch.selection controls which learner is used.\n\nas.data.table(graph_learner$param_set)\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n removeconstants_prerobustify.ratio \n ParamDbl \n 0 \n 1 \n Inf \n \n \n removeconstants_prerobustify.rel_tol \n ParamDbl \n 0 \n Inf \n Inf \n \n \n removeconstants_prerobustify.abs_tol \n ParamDbl \n 0 \n Inf \n Inf \n \n \n removeconstants_prerobustify.na_ignore \n ParamLgl \n NA \n NA \n 2 \n \n \n removeconstants_prerobustify.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n imputehist.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n missind.which \n ParamFct \n NA \n NA \n 2 \n \n \n missind.type \n ParamFct \n NA \n NA \n 4 \n \n \n missind.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n imputesample.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n encode.method \n ParamFct \n NA \n NA \n 5 \n \n \n encode.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n removeconstants_postrobustify.ratio \n ParamDbl \n 0 \n 1 \n Inf \n \n \n removeconstants_postrobustify.rel_tol \n ParamDbl \n 0 \n Inf \n Inf \n \n \n removeconstants_postrobustify.abs_tol \n ParamDbl \n 0 \n Inf \n Inf \n \n \n removeconstants_postrobustify.na_ignore \n ParamLgl \n NA \n NA \n 2 \n \n \n removeconstants_postrobustify.affect_columns \n ParamUty \n NA \n NA \n Inf \n \n \n kknn.k \n ParamInt \n 1 \n Inf \n Inf \n \n \n kknn.distance \n ParamDbl \n 0 \n Inf \n Inf \n \n \n kknn.kernel \n ParamFct \n NA \n NA \n 10 \n \n \n kknn.scale \n ParamLgl \n NA \n NA \n 2 \n \n \n kknn.ykernel \n ParamUty \n NA \n NA \n Inf \n \n \n kknn.store_model \n ParamLgl \n NA \n NA \n 2 \n \n \n svm.cachesize \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n svm.class.weights \n ParamUty \n NA \n NA \n Inf \n \n \n svm.coef0 \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n svm.cost \n ParamDbl \n 0 \n Inf \n Inf \n \n \n svm.cross \n ParamInt \n 0 \n Inf \n Inf \n \n \n svm.decision.values \n ParamLgl \n NA \n NA \n 2 \n \n \n svm.degree \n ParamInt \n 1 \n Inf \n Inf \n \n \n svm.epsilon \n ParamDbl \n 0 \n Inf \n Inf \n \n \n svm.fitted \n ParamLgl \n NA \n NA \n 2 \n \n \n svm.gamma \n ParamDbl \n 0 \n Inf \n Inf \n \n \n svm.kernel \n ParamFct \n NA \n NA \n 4 \n \n \n svm.nu \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n svm.scale \n ParamUty \n NA \n NA \n Inf \n \n \n svm.shrinking \n ParamLgl \n NA \n NA \n 2 \n \n \n svm.tolerance \n ParamDbl \n 0 \n Inf \n Inf \n \n \n svm.type \n ParamFct \n NA \n NA \n 2 \n \n \n ranger.alpha \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n ranger.always.split.variables \n ParamUty \n NA \n NA \n Inf \n \n \n ranger.class.weights \n ParamUty \n NA \n NA \n Inf \n \n \n ranger.holdout \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.importance \n ParamFct \n NA \n NA \n 4 \n \n \n ranger.keep.inbag \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.max.depth \n ParamInt \n 0 \n Inf \n Inf \n \n \n ranger.min.node.size \n ParamInt \n 1 \n Inf \n Inf \n \n \n ranger.min.prop \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n ranger.minprop \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n ranger.mtry \n ParamInt \n 1 \n Inf \n Inf \n \n \n ranger.mtry.ratio \n ParamDbl \n 0 \n 1 \n Inf \n \n \n ranger.num.random.splits \n ParamInt \n 1 \n Inf \n Inf \n \n \n ranger.num.threads \n ParamInt \n 1 \n Inf \n Inf \n \n \n ranger.num.trees \n ParamInt \n 1 \n Inf \n Inf \n \n \n ranger.oob.error \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.regularization.factor \n ParamUty \n NA \n NA \n Inf \n \n \n ranger.regularization.usedepth \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.replace \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.respect.unordered.factors \n ParamFct \n NA \n NA \n 3 \n \n \n ranger.sample.fraction \n ParamDbl \n 0 \n 1 \n Inf \n \n \n ranger.save.memory \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.scale.permutation.importance \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.se.method \n ParamFct \n NA \n NA \n 2 \n \n \n ranger.seed \n ParamInt \n -Inf \n Inf \n Inf \n \n \n ranger.split.select.weights \n ParamUty \n NA \n NA \n Inf \n \n \n ranger.splitrule \n ParamFct \n NA \n NA \n 3 \n \n \n ranger.verbose \n ParamLgl \n NA \n NA \n 2 \n \n \n ranger.write.forest \n ParamLgl \n NA \n NA \n 2 \n \n \n branch.selection \n ParamFct \n NA \n NA \n 3 \n \n\n\n\n\n\n\nTune the pipeline\nWe will only tune one hyperparameter for each learner in this example. Additionally, we tune the branching parameter which selects one of the three learners. We have to specify that a hyperparameter is only valid for a certain learner by using depends = branch.selection == .\n\n# branch\ngraph_learner$param_set$values$branch.selection =\n to_tune(c(\"kknn\", \"svm\", \"ranger\"))\n\n# kknn\ngraph_learner$param_set$values$kknn.k =\n to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == \"kknn\"))\n\n# svm\ngraph_learner$param_set$values$svm.cost =\n to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == \"svm\"))\n\n# ranger\ngraph_learner$param_set$values$ranger.mtry =\n to_tune(p_int(1, 8, depends = branch.selection == \"ranger\"))\n\n# short learner id for printing\ngraph_learner$id = \"graph_learner\"\n\nWe define a tuning instance and select a random search which is stopped after 20 evaluated configurations.\n\ninstance = tune(\n method = \"random_search\",\n task = task,\n learner = graph_learner,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 20\n)\n\n\n\n\n\n\n\nThe following shows a quick way to visualize the tuning results.\n\nautoplot(instance, type = \"marginal\",\n cols_x = c(\"x_domain_kknn.k\",\"x_domain_svm.cost\", \"ranger.mtry\"))\n\n\n\n\n\n\n\n\n\n\nFinal Model\nWe add the optimized hyperparameters to the graph learner and train the learner on the full dataset.\n\nlearner = as_learner(graph)\nlearner$param_set$values = instance$result_learner_param_vals\nlearner$train(task)\n\nThe trained model can now be used to make predictions on new data. A common mistake is to report the performance estimated on the resampling sets on which the tuning was performed (instance$result_y) as the model’s performance. Instead we have to use nested resampling to get an unbiased performance estimate.\n\n\nNested Resampling\nWe use nested resampling to get an unbiased estimate of the predictive performance of our graph learner.\n\ngraph_learner = as_learner(graph)\ngraph_learner$param_set$values$branch.selection =\n to_tune(c(\"kknn\", \"svm\", \"ranger\"))\ngraph_learner$param_set$values$kknn.k =\n to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == \"kknn\"))\ngraph_learner$param_set$values$svm.cost =\n to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == \"svm\"))\ngraph_learner$param_set$values$ranger.mtry =\n to_tune(p_int(1, 8, depends = branch.selection == \"ranger\"))\ngraph_learner$id = \"graph_learner\"\n\ninner_resampling = rsmp(\"cv\", folds = 3)\nat = AutoTuner$new(\n learner = graph_learner,\n resampling = inner_resampling,\n measure = msr(\"classif.ce\"),\n terminator = trm(\"evals\", n_evals = 10),\n tuner = tnr(\"random_search\")\n)\n\nouter_resampling = rsmp(\"cv\", folds = 3)\nrr = resample(task, at, outer_resampling, store_models = TRUE)\n\n\n\n\n\n\n\nWe check the inner tuning results for stable hyperparameters. This means that the selected hyperparameters should not vary too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduce too much randomness. Usually, we aim for the selection of stable hyperparameters for all outer training sets.\n\nextract_inner_tuning_results(rr)\n\n\n\n\n\n\n\n\nNext, we want to compare the predictive performances estimated on the outer resampling to the inner resampling. Significantly lower predictive performances on the outer resampling indicate that the models with the optimized hyperparameters overfit the data.\n\nrr$score()\n\n\n\n\n\n \n \n iteration \n task_id \n learner_id \n resampling_id \n classif.ce \n \n \n\n \n 1 \n pima \n graph_learner.tuned \n cv \n 0.2304688 \n \n \n 2 \n pima \n graph_learner.tuned \n cv \n 0.2539062 \n \n \n 3 \n pima \n graph_learner.tuned \n cv \n 0.2109375 \n \n\n\n\n\nThe aggregated performance of all outer resampling iterations is essentially the unbiased performance of the graph learner with optimal hyperparameter found by random search.\n\nrr$aggregate()\n\nclassif.ce \n 0.2317708 \n\n\nApplying nested resampling can be shortened by using the tune_nested()-shortcut.\n\ngraph_learner = as_learner(graph)\ngraph_learner$param_set$values$branch.selection =\n to_tune(c(\"kknn\", \"svm\", \"ranger\"))\ngraph_learner$param_set$values$kknn.k =\n to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == \"kknn\"))\ngraph_learner$param_set$values$svm.cost =\n to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == \"svm\"))\ngraph_learner$param_set$values$ranger.mtry =\n to_tune(p_int(1, 8, depends = branch.selection == \"ranger\"))\ngraph_learner$id = \"graph_learner\"\n\nrr = tune_nested(\n method = \"random_search\",\n task = task,\n learner = graph_learner,\n inner_resampling = rsmp (\"cv\", folds = 3),\n outer_resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 10,\n)\n\n\n\nResources\nThe mlr3book includes chapters on pipelines and hyperparameter tuning. The mlr3cheatsheets contain frequently used commands and workflows of mlr3." - }, - { - "objectID": "gallery/2021-03-09-practical-tuning-series-tune-a-support-vector-machine/practical-tuning-series-tune-a-support-vector-machine.html", - "href": "gallery/2021-03-09-practical-tuning-series-tune-a-support-vector-machine/practical-tuning-series-tune-a-support-vector-machine.html", - "title": "Practical Tuning Series - Tune a Support Vector Machine", - "section": "", - "text": "Scope\nThis is the first part of the practical tuning series. The other parts can be found here:\n\nPart II - Tune a Preprocessing Pipeline\nPart III - Build an Automated Machine Learning System\nPart IV - Tuning and Parallel Processing\n\nIn this post, we demonstrate how to optimize the hyperparameters of a support vector machine (SVM). We are using the mlr3 machine learning framework with the mlr3tuning extension package.\nFirst, we start by showing the basic building blocks of mlr3tuning and tune the cost and gamma hyperparameters of an SVM with a radial basis function on the Iris data set. After that, we use transformations to tune the both hyperparameters on the logarithmic scale. Next, we explain the importance of dependencies to tune hyperparameters like degree which are dependent on the choice of kernel. After that, we fit an SVM with optimized hyperparameters on the full dataset. Finally, nested resampling is used to compute an unbiased performance estimate of our tuned SVM.\n\n\nPrerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented. The lgr package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning ).\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nIn the example, we use the Iris data set which classifies 150 flowers in three species of Iris. The flowers are characterized by sepal length and width and petal length and width. The Iris data set allows us to quickly fit models to it. However, the influence of hyperparameter tuning on the predictive performance might be minor. Other data sets might give more meaningful tuning results.\n\n# retrieve the task from mlr3\ntask = tsk(\"iris\")\n\n# generate a quick textual overview using the skimr package\nskimr::skim(task$data())\n\n\nData summary\n\n\nName\ntask$data()\n\n\nNumber of rows\n150\n\n\nNumber of columns\n5\n\n\nKey\nNULL\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n1\n\n\nnumeric\n4\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\nSpecies\n0\n1\nFALSE\n3\nset: 50, ver: 50, vir: 50\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nPetal.Length\n0\n1\n3.76\n1.77\n1.0\n1.6\n4.35\n5.1\n6.9\n▇▁▆▇▂\n\n\nPetal.Width\n0\n1\n1.20\n0.76\n0.1\n0.3\n1.30\n1.8\n2.5\n▇▁▇▅▃\n\n\nSepal.Length\n0\n1\n5.84\n0.83\n4.3\n5.1\n5.80\n6.4\n7.9\n▆▇▇▅▂\n\n\nSepal.Width\n0\n1\n3.06\n0.44\n2.0\n2.8\n3.00\n3.3\n4.4\n▁▆▇▂▁\n\n\n\n\n\nWe choose the support vector machine implementation from the e1071 package (which is based on LIBSVM) and use it as a classification machine by setting type to \"C-classification\".\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\", kernel = \"radial\")\n\n\n\nTuning Search Space\nFor tuning, it is important to create a search space that defines the type and range of the hyperparameters. A learner stores all information about its hyperparameters in the slot $param_set. Not all parameters are tunable. We have to choose a subset of the hyperparameters we want to tune.\n\nas.data.table(learner$param_set)\n\n\n\n\n\n \n \n id \n class \n lower \n upper \n nlevels \n \n \n\n \n cachesize \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n class.weights \n ParamUty \n NA \n NA \n Inf \n \n \n coef0 \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n cost \n ParamDbl \n 0 \n Inf \n Inf \n \n \n cross \n ParamInt \n 0 \n Inf \n Inf \n \n \n decision.values \n ParamLgl \n NA \n NA \n 2 \n \n \n degree \n ParamInt \n 1 \n Inf \n Inf \n \n \n epsilon \n ParamDbl \n 0 \n Inf \n Inf \n \n \n fitted \n ParamLgl \n NA \n NA \n 2 \n \n \n gamma \n ParamDbl \n 0 \n Inf \n Inf \n \n \n kernel \n ParamFct \n NA \n NA \n 4 \n \n \n nu \n ParamDbl \n -Inf \n Inf \n Inf \n \n \n scale \n ParamUty \n NA \n NA \n Inf \n \n \n shrinking \n ParamLgl \n NA \n NA \n 2 \n \n \n tolerance \n ParamDbl \n 0 \n Inf \n Inf \n \n \n type \n ParamFct \n NA \n NA \n 2 \n \n\n\n\n\nWe use the to_tune() function to define the range over which the hyperparameter should be tuned. We opt for the cost and gamma hyperparameters of the radial kernel and set the tuning ranges with lower and upper bounds.\n\nlearner$param_set$values$cost = to_tune(0.1, 10)\nlearner$param_set$values$gamma = to_tune(0, 5)\n\n\n\nTuning\nWe specify how to evaluate the performance of the different hyperparameter configurations. For this, we choose 3-fold cross validation as the resampling strategy and the classification error as the performance measure.\n\nresampling = rsmp(\"cv\", folds = 3)\nmeasure = msr(\"classif.ce\")\n\nUsually, we have to select a budget for the tuning. This is done by choosing a Terminator, which stops the tuning e.g. after a performance level is reached or after a given time. However, some tuners like grid search terminate themselves. In this case, we choose a terminator that never stops and the tuning is not stopped before all grid points are evaluated.\n\nterminator = trm(\"none\")\n\nAt this point, we can construct a TuningInstanceSingleCrit that describes the tuning problem.\n\ninstance = TuningInstanceSingleCrit$new(\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n terminator = terminator\n)\n\nprint(instance)\n\n\n* State: Not optimized\n* Objective: \n* Search Space:\n id class lower upper nlevels\n1: cost ParamDbl 0.1 10 Inf\n2: gamma ParamDbl 0.0 5 Inf\n* Terminator: \n\n\nFinally, we have to choose a Tuner. Grid Search discretizes numeric parameters into a given resolution and constructs a grid from the Cartesian product of these sets. Categorical parameters produce a grid over all levels specified in the search space. In this example, we only use a resolution of 5 to keep the runtime low. Usually, a higher resolution is used to create a denser grid.\n\ntuner = tnr(\"grid_search\", resolution = 5)\n\nprint(tuner)\n\n: Grid Search\n* Parameters: resolution=5, batch_size=1\n* Parameter classes: ParamLgl, ParamInt, ParamDbl, ParamFct\n* Properties: dependencies, single-crit, multi-crit\n* Packages: mlr3tuning\n\n\nWe can preview the proposed configurations by using generate_design_grid(). This function is internally executed by TunerGridSearch.\n\ngenerate_design_grid(learner$param_set$search_space(), resolution = 5)\n\n with 25 rows:\n cost gamma\n 1: 0.100 0.00\n 2: 0.100 1.25\n 3: 0.100 2.50\n 4: 0.100 3.75\n 5: 0.100 5.00\n 6: 2.575 0.00\n 7: 2.575 1.25\n 8: 2.575 2.50\n 9: 2.575 3.75\n10: 2.575 5.00\n11: 5.050 0.00\n12: 5.050 1.25\n13: 5.050 2.50\n14: 5.050 3.75\n15: 5.050 5.00\n16: 7.525 0.00\n17: 7.525 1.25\n18: 7.525 2.50\n19: 7.525 3.75\n20: 7.525 5.00\n21: 10.000 0.00\n22: 10.000 1.25\n23: 10.000 2.50\n24: 10.000 3.75\n25: 10.000 5.00\n cost gamma\n\n\nWe trigger the tuning by passing the TuningInstanceSingleCrit to the $optimize() method of the Tuner. The instance is modified in-place.\n\ntuner$optimize(instance)\n\n cost gamma learner_param_vals x_domain classif.ce\n1: 5.05 1.25 0.04\n\n\n\n\n\n\n\n\nWe plot the performances depending on the evaluated cost and gamma values.\n\nautoplot(instance, type = \"surface\", cols_x = c(\"cost\", \"gamma\"),\n learner = lrn(\"regr.km\"))\n\n\n\n\n\n\n\n\n\n\nThe points mark the evaluated cost and gamma values. We should not infer the performance of new values from the heatmap since it is only an interpolation. However, we can see the general interaction between the hyperparameters.\nTuning a learner can be shortened by using the tune()-shortcut.\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\", kernel = \"radial\")\nlearner$param_set$values$cost = to_tune(0.1, 10)\nlearner$param_set$values$gamma = to_tune(0, 5)\n\ninstance = tune(\n method = \"grid_search\",\n task = tsk(\"iris\"),\n learner = learner,\n resampling = rsmp (\"holdout\"),\n measure = msr(\"classif.ce\"),\n resolution = 5\n)\n\n\n\nTransformation\nNext, we want to tune the cost and gamma hyperparameter more efficiently. It is recommended to tune cost and gamma on the logarithmic scale (Hsu, Chang, and Lin 2003). The log transformation emphasizes smaller cost and gamma values but also creates large values. Therefore, we use a log transformation to emphasize this region of the search space with a denser grid.\nGenerally speaking, transformations can be used to convert hyperparameters to a new scale. These transformations are applied before the proposed configuration is passed to the Learner. We can directly define the transformation in the to_tune() function. The lower and upper bound is set on the original scale.\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\", kernel = \"radial\")\n\n# tune from 2^-15 to 2^15 on a log scale\nlearner$param_set$values$cost = to_tune(p_dbl(-15, 15, trafo = function(x) 2^x))\n\n# tune from 2^-15 to 2^5 on a log scale\nlearner$param_set$values$gamma = to_tune(p_dbl(-15, 5, trafo = function(x) 2^x))\n\nTransformations to the log scale are the ones most commonly used. We can use a shortcut for this transformation. The lower and upper bound is set on the transformed scale.\n\nlearner$param_set$values$cost = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$gamma = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\n\nWe use the tune()-shortcut to run the tuning.\n\ninstance = tune(\n method = \"grid_search\",\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n resolution = 5\n)\n\n\n\n\n\n\n\nThe hyperparameter values after the transformation are stored in the x_domain column as lists. We can expand these lists into multiple columns by using as.data.table(). The hyperparameter names are prefixed by x_domain.\n\nas.data.table(instance$archive)\n\n\n\n\n\n \n \n cost \n gamma \n x_domain_cost \n x_domain_gamma \n \n \n\n \n 11.512925 \n -11.512925 \n 1.000000e+05 \n 1.000000e-05 \n \n \n 5.756463 \n 0.000000 \n 3.162278e+02 \n 1.000000e+00 \n \n \n -11.512925 \n 11.512925 \n 1.000000e-05 \n 1.000000e+05 \n \n \n 0.000000 \n 5.756463 \n 1.000000e+00 \n 3.162278e+02 \n \n \n -11.512925 \n -5.756463 \n 1.000000e-05 \n 3.162300e-03 \n \n \n 0.000000 \n 0.000000 \n 1.000000e+00 \n 1.000000e+00 \n \n \n 11.512925 \n 5.756463 \n 1.000000e+05 \n 3.162278e+02 \n \n \n -5.756463 \n -11.512925 \n 3.162300e-03 \n 1.000000e-05 \n \n \n -11.512925 \n -11.512925 \n 1.000000e-05 \n 1.000000e-05 \n \n \n -5.756463 \n 11.512925 \n 3.162300e-03 \n 1.000000e+05 \n \n \n -11.512925 \n 5.756463 \n 1.000000e-05 \n 3.162278e+02 \n \n \n 11.512925 \n 0.000000 \n 1.000000e+05 \n 1.000000e+00 \n \n \n -11.512925 \n 0.000000 \n 1.000000e-05 \n 1.000000e+00 \n \n \n 5.756463 \n -11.512925 \n 3.162278e+02 \n 1.000000e-05 \n \n \n 5.756463 \n 5.756463 \n 3.162278e+02 \n 3.162278e+02 \n \n \n 5.756463 \n -5.756463 \n 3.162278e+02 \n 3.162300e-03 \n \n \n 5.756463 \n 11.512925 \n 3.162278e+02 \n 1.000000e+05 \n \n \n 11.512925 \n 11.512925 \n 1.000000e+05 \n 1.000000e+05 \n \n \n 11.512925 \n -5.756463 \n 1.000000e+05 \n 3.162300e-03 \n \n \n -5.756463 \n -5.756463 \n 3.162300e-03 \n 3.162300e-03 \n \n \n 0.000000 \n -11.512925 \n 1.000000e+00 \n 1.000000e-05 \n \n \n 0.000000 \n 11.512925 \n 1.000000e+00 \n 1.000000e+05 \n \n \n 0.000000 \n -5.756463 \n 1.000000e+00 \n 3.162300e-03 \n \n \n -5.756463 \n 0.000000 \n 3.162300e-03 \n 1.000000e+00 \n \n \n -5.756463 \n 5.756463 \n 3.162300e-03 \n 3.162278e+02 \n \n\n\n\n\nWe plot the performances depending on the evaluated cost and gamma values.\n\nlibrary(ggplot2)\nlibrary(scales)\nautoplot(instance, type = \"points\", cols_x = c(\"x_domain_cost\", \"x_domain_gamma\")) +\n scale_x_continuous(\n trans = log2_trans(),\n breaks = trans_breaks(\"log10\", function(x) 10^x),\n labels = trans_format(\"log10\", math_format(10^.x))) +\n scale_y_continuous(\n trans = log2_trans(),\n breaks = trans_breaks(\"log10\", function(x) 10^x),\n labels = trans_format(\"log10\", math_format(10^.x)))\n\n\n\n\n\n\n\n\n\n\nDependencies\nDependencies ensure that certain parameters are only proposed depending on values of other hyperparameters. We want to tune the degree hyperparameter that is only needed for the polynomial kernel.\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\")\n\nlearner$param_set$values$cost = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$gamma = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\n\nlearner$param_set$values$kernel = to_tune(c(\"polynomial\", \"radial\"))\nlearner$param_set$values$degree = to_tune(1, 4)\n\nThe dependencies are already stored in the learner parameter set.\n\nlearner$param_set$deps\n\n id on cond\n1: cost type \n2: nu type \n3: degree kernel \n4: coef0 kernel \n5: gamma kernel \n\n\nThe gamma hyperparameter depends on the kernel being polynomial, radial or sigmoid\n\nlearner$param_set$deps$cond[[5]]\n\nCondAnyOf: x ∈ {polynomial, radial, sigmoid}\n\n\nwhereas the degree hyperparameter is solely used by the polynomial kernel.\n\nlearner$param_set$deps$cond[[3]]\n\nCondEqual: x = polynomial\n\n\nWe preview the grid to show the effect of the dependencies.\n\ngenerate_design_grid(learner$param_set$search_space(), resolution = 2)\n\n with 12 rows:\n cost gamma kernel degree\n 1: -11.51293 -11.51293 polynomial 1\n 2: -11.51293 -11.51293 polynomial 4\n 3: -11.51293 -11.51293 radial NA\n 4: -11.51293 11.51293 polynomial 1\n 5: -11.51293 11.51293 polynomial 4\n 6: -11.51293 11.51293 radial NA\n 7: 11.51293 -11.51293 polynomial 1\n 8: 11.51293 -11.51293 polynomial 4\n 9: 11.51293 -11.51293 radial NA\n10: 11.51293 11.51293 polynomial 1\n11: 11.51293 11.51293 polynomial 4\n12: 11.51293 11.51293 radial NA\n\n\nThe value for degree is NA if the dependency on the kernel is not satisfied.\nWe use the tune()-shortcut to run the tuning.\n\ninstance = tune(\n method = \"grid_search\",\n task = task,\n learner = learner,\n resampling = resampling,\n measure = measure,\n resolution = 3\n)\n\n\n\n\n\n\n\n\ninstance$result\n\n cost gamma kernel degree learner_param_vals x_domain classif.ce\n1: 0 0 polynomial 1 0.02\n\n\n\n\nFinal Model\nWe add the optimized hyperparameters to the learner and train the learner on the full dataset.\n\nlearner = lrn(\"classif.svm\")\nlearner$param_set$values = instance$result_learner_param_vals\nlearner$train(task)\n\nThe trained model can now be used to make predictions on new data. A common mistake is to report the performance estimated on the resampling sets on which the tuning was performed (instance$result_y) as the model’s performance. These scores might be biased and overestimate the ability of the fitted model to predict with new data. Instead, we have to use nested resampling to get an unbiased performance estimate.\n\n\nNested Resampling\nTuning should not be performed on the same resampling sets which are used for evaluating the model itself, since this would result in a biased performance estimate. Nested resampling uses an outer and inner resampling to separate the tuning from the performance estimation of the model. We can use the AutoTuner class for running nested resampling. The AutoTuner wraps a Learner and tunes the hyperparameter of the learner during $train(). This is our inner resampling loop.\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\")\nlearner$param_set$values$cost = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$gamma = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$kernel = to_tune(c(\"polynomial\", \"radial\"))\nlearner$param_set$values$degree = to_tune(1, 4)\n\nresampling_inner = rsmp(\"cv\", folds = 3)\nterminator = trm(\"none\")\ntuner = tnr(\"grid_search\", resolution = 3)\n\nat = AutoTuner$new(\n learner = learner,\n resampling = resampling_inner,\n measure = measure,\n terminator = terminator,\n tuner = tuner,\n store_models = TRUE)\n\nWe put the AutoTuner into a resample() call to get the outer resampling loop.\n\nresampling_outer = rsmp(\"cv\", folds = 3)\nrr = resample(task = task, learner = at, resampling = resampling_outer, store_models = TRUE)\n\n\n\n\n\n\n\nWe check the inner tuning results for stable hyperparameters. This means that the selected hyperparameters should not vary too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduce too much randomness. Usually, we aim for the selection of stable hyperparameters for all outer training sets.\n\nextract_inner_tuning_results(rr)\n\n\n\n\n\n \n \n iteration \n cost \n gamma \n kernel \n degree \n classif.ce \n task_id \n learner_id \n resampling_id \n \n \n\n \n 1 \n 0.00000 \n 11.51293 \n polynomial \n 1 \n 0.0401070 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n -11.51293 \n radial \n NA \n 0.0496138 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n -11.51293 \n radial \n NA \n 0.0303030 \n iris \n classif.svm.tuned \n cv \n \n\n\n\n\nNext, we want to compare the predictive performances estimated on the outer resampling to the inner resampling (extract_inner_tuning_results(rr)). Significantly lower predictive performances on the outer resampling indicate that the models with the optimized hyperparameters overfit the data.\n\nrr$score()\n\n\n\n\n\n \n \n iteration \n task_id \n learner_id \n resampling_id \n classif.ce \n \n \n\n \n 1 \n iris \n classif.svm.tuned \n cv \n 0.06 \n \n \n 2 \n iris \n classif.svm.tuned \n cv \n 0.04 \n \n \n 3 \n iris \n classif.svm.tuned \n cv \n 0.04 \n \n\n\n\n\nThe archives of the AutoTuners allows us to inspect all evaluated hyperparameters configurations with the associated predictive performances.\n\nextract_inner_tuning_archives(rr)\n\n\n\n\n\n \n \n iteration \n cost \n gamma \n kernel \n degree \n classif.ce \n runtime_learners \n batch_nr \n warnings \n errors \n task_id \n learner_id \n resampling_id \n \n \n\n \n 1 \n 11.51293 \n 11.51293 \n polynomial \n 2 \n 0.1705288 \n 0.016 \n 1 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n -11.51293 \n polynomial \n 1 \n 0.5392157 \n 0.009 \n 2 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 11.51293 \n radial \n NA \n 0.6200238 \n 0.007 \n 3 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 0.00000 \n polynomial \n 4 \n 0.1209150 \n 0.008 \n 4 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 0.00000 \n radial \n NA \n 0.0704100 \n 0.009 \n 5 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 0.00000 \n polynomial \n 1 \n 0.5392157 \n 0.009 \n 6 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n -11.51293 \n radial \n NA \n 0.5392157 \n 0.007 \n 7 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n -11.51293 \n polynomial \n 4 \n 0.5998217 \n 0.006 \n 8 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 0.00000 \n radial \n NA \n 0.5392157 \n 0.005 \n 9 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 0.00000 \n polynomial \n 1 \n 0.0502080 \n 0.005 \n 10 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0502080 \n 0.007 \n 11 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 0.00000 \n polynomial \n 2 \n 0.1313131 \n 0.007 \n 12 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 0.00000 \n polynomial \n 4 \n 0.1408200 \n 0.007 \n 13 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n -11.51293 \n polynomial \n 1 \n 0.5392157 \n 0.006 \n 14 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n -11.51293 \n polynomial \n 4 \n 0.6705288 \n 0.008 \n 15 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 11.51293 \n polynomial \n 1 \n 0.0401070 \n 0.028 \n 16 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n -11.51293 \n polynomial \n 1 \n 0.0502080 \n 0.005 \n 17 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1408200 \n 0.006 \n 18 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 0.00000 \n polynomial \n 2 \n 0.5392157 \n 0.006 \n 19 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 0.00000 \n polynomial \n 1 \n 0.0401070 \n 0.023 \n 20 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 11.51293 \n polynomial \n 4 \n 0.1408200 \n 0.008 \n 21 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n -11.51293 \n polynomial \n 4 \n 0.6705288 \n 0.007 \n 22 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 0.00000 \n polynomial \n 2 \n 0.1705288 \n 0.013 \n 23 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 0.00000 \n radial \n NA \n 0.0805110 \n 0.004 \n 24 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0989305 \n 0.720 \n 25 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 11.51293 \n polynomial \n 2 \n 0.1705288 \n 0.011 \n 26 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n 11.51293 \n radial \n NA \n 0.5998217 \n 0.007 \n 27 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n -11.51293 \n radial \n NA \n 0.5392157 \n 0.006 \n 28 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 11.51293 \n radial \n NA \n 0.5998217 \n 0.007 \n 29 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n 11.51293 \n polynomial \n 2 \n 0.1705288 \n 0.013 \n 30 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n -11.51293 \n polynomial \n 2 \n 0.5392157 \n 0.006 \n 31 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 0.00000 \n -11.51293 \n polynomial \n 2 \n 0.5392157 \n 0.006 \n 32 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1408200 \n 0.008 \n 33 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n -11.51293 \n polynomial \n 2 \n 0.5392157 \n 0.008 \n 34 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n 11.51293 \n -11.51293 \n radial \n NA \n 0.0404040 \n 0.005 \n 35 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 1 \n -11.51293 \n 0.00000 \n polynomial \n 4 \n 0.5998217 \n 0.007 \n 36 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 0.00000 \n polynomial \n 2 \n 0.1883541 \n 0.012 \n 1 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 0.00000 \n polynomial \n 4 \n 0.1892454 \n 0.025 \n 2 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n -11.51293 \n polynomial \n 2 \n 0.7608437 \n 0.007 \n 3 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n -11.51293 \n polynomial \n 1 \n 0.7608437 \n 0.008 \n 4 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 0.00000 \n polynomial \n 4 \n 0.7608437 \n 0.008 \n 5 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 0.00000 \n polynomial \n 1 \n 0.7608437 \n 0.010 \n 6 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 0.00000 \n polynomial \n 4 \n 0.1892454 \n 0.008 \n 7 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n -11.51293 \n polynomial \n 1 \n 0.7608437 \n 0.007 \n 8 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 11.51293 \n polynomial \n 2 \n 0.2376708 \n 1.048 \n 9 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 11.51293 \n polynomial \n 4 \n 0.1892454 \n 0.024 \n 10 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 0.00000 \n radial \n NA \n 0.0799168 \n 0.006 \n 11 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 11.51293 \n polynomial \n 2 \n 0.2376708 \n 1.039 \n 12 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n -11.51293 \n radial \n NA \n 0.0496138 \n 0.006 \n 13 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 11.51293 \n radial \n NA \n 0.7608437 \n 0.006 \n 14 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 0.00000 \n polynomial \n 2 \n 0.7608437 \n 0.007 \n 15 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 11.51293 \n radial \n NA \n 0.7608437 \n 0.011 \n 16 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1892454 \n 0.023 \n 17 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0799168 \n 0.005 \n 18 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 0.00000 \n polynomial \n 1 \n 0.0496138 \n 0.007 \n 19 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 11.51293 \n polynomial \n 1 \n 0.0799168 \n 0.007 \n 20 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n -11.51293 \n polynomial \n 2 \n 0.7608437 \n 0.004 \n 21 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n -11.51293 \n radial \n NA \n 0.7608437 \n 0.007 \n 22 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0496138 \n 0.006 \n 23 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n -11.51293 \n polynomial \n 4 \n 0.7608437 \n 0.009 \n 24 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 0.00000 \n polynomial \n 2 \n 0.1494355 \n 0.758 \n 25 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n 0.00000 \n radial \n NA \n 0.0600119 \n 0.007 \n 26 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 0.00000 \n -11.51293 \n polynomial \n 4 \n 0.7608437 \n 0.009 \n 27 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n -11.51293 \n polynomial \n 1 \n 0.0496138 \n 0.006 \n 28 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n 0.00000 \n polynomial \n 1 \n 0.0799168 \n 0.005 \n 29 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 11.51293 \n polynomial \n 2 \n 0.1592395 \n 0.828 \n 30 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 0.00000 \n radial \n NA \n 0.7608437 \n 0.008 \n 31 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 11.51293 \n radial \n NA \n 0.7608437 \n 0.013 \n 32 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n -11.51293 \n polynomial \n 2 \n 0.7608437 \n 0.004 \n 33 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n -11.51293 \n radial \n NA \n 0.7608437 \n 0.006 \n 34 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n 11.51293 \n -11.51293 \n polynomial \n 4 \n 0.7608437 \n 0.005 \n 35 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 2 \n -11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1892454 \n 0.024 \n 36 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 0.00000 \n polynomial \n 1 \n 0.0597148 \n 0.009 \n 1 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 0.00000 \n polynomial \n 1 \n 0.7192513 \n 0.007 \n 2 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n -11.51293 \n polynomial \n 2 \n 0.7192513 \n 0.007 \n 3 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 0.00000 \n polynomial \n 2 \n 0.0900178 \n 0.007 \n 4 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n -11.51293 \n polynomial \n 2 \n 0.7192513 \n 0.007 \n 5 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 0.00000 \n polynomial \n 2 \n 0.0802139 \n 0.007 \n 6 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 11.51293 \n radial \n NA \n 0.7192513 \n 0.007 \n 7 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 0.00000 \n polynomial \n 4 \n 0.7192513 \n 0.011 \n 8 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 11.51293 \n polynomial \n 1 \n 0.0597148 \n 0.005 \n 9 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n -11.51293 \n polynomial \n 4 \n 0.7192513 \n 0.007 \n 10 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n -11.51293 \n polynomial \n 2 \n 0.7192513 \n 0.008 \n 11 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n -11.51293 \n polynomial \n 4 \n 0.7192513 \n 0.008 \n 12 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n -11.51293 \n polynomial \n 1 \n 0.0404040 \n 0.007 \n 13 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 0.00000 \n radial \n NA \n 0.7192513 \n 0.009 \n 14 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n -11.51293 \n radial \n NA \n 0.7192513 \n 0.006 \n 15 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 0.00000 \n radial \n NA \n 0.0897207 \n 0.007 \n 16 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 0.00000 \n radial \n NA \n 0.0505051 \n 0.005 \n 17 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0404040 \n 0.007 \n 18 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n -11.51293 \n polynomial \n 1 \n 0.7192513 \n 0.009 \n 19 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n -11.51293 \n polynomial \n 1 \n 0.7192513 \n 0.006 \n 20 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 11.51293 \n polynomial \n 2 \n 0.0900178 \n 0.006 \n 21 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 0.00000 \n polynomial \n 1 \n 0.0404040 \n 0.007 \n 22 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 0.00000 \n polynomial \n 4 \n 0.1589424 \n 0.008 \n 23 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 11.51293 \n polynomial \n 1 \n 0.0597148 \n 0.011 \n 24 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1488414 \n 0.006 \n 25 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n -11.51293 \n polynomial \n 4 \n 0.7192513 \n 0.006 \n 26 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 11.51293 \n polynomial \n 2 \n 0.0900178 \n 0.008 \n 27 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 11.51293 \n polynomial \n 4 \n 0.1488414 \n 0.007 \n 28 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 11.51293 \n radial \n NA \n 0.7192513 \n 0.005 \n 29 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n -11.51293 \n radial \n NA \n 0.7192513 \n 0.006 \n 30 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 11.51293 \n radial \n NA \n 0.7192513 \n 0.007 \n 31 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n 0.00000 \n polynomial \n 4 \n 0.1488414 \n 0.008 \n 32 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 11.51293 \n polynomial \n 4 \n 0.1488414 \n 0.008 \n 33 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 11.51293 \n -11.51293 \n radial \n NA \n 0.0303030 \n 0.007 \n 34 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n -11.51293 \n 0.00000 \n polynomial \n 2 \n 0.7192513 \n 0.008 \n 35 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n \n 3 \n 0.00000 \n 11.51293 \n polynomial \n 2 \n 0.0900178 \n 0.008 \n 36 \n 0 \n 0 \n iris \n classif.svm.tuned \n cv \n \n\n\n\n\nThe aggregated performance of all outer resampling iterations is essentially the unbiased performance of an SVM with optimal hyperparameter found by grid search.\n\nrr$aggregate()\n\nclassif.ce \n0.04666667 \n\n\nApplying nested resampling can be shortened by using the tune_nested()-shortcut.\n\nlearner = lrn(\"classif.svm\", type = \"C-classification\")\nlearner$param_set$values$cost = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$gamma = to_tune(p_dbl(1e-5, 1e5, logscale = TRUE))\nlearner$param_set$values$kernel = to_tune(c(\"polynomial\", \"radial\"))\nlearner$param_set$values$degree = to_tune(1, 4)\n\nrr = tune_nested(\n method = \"grid_search\",\n task = tsk(\"iris\"),\n learner = learner,\n inner_resampling = rsmp (\"cv\", folds = 3),\n outer_resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n resolution = 3\n)\n\n\n\nResources\nThe mlr3book includes chapters on tuning spaces and hyperparameter tuning. The mlr3cheatsheets contain frequently used commands and workflows of mlr3.\n\n\n\n\n\nReferences\n\nHsu, Chih-wei, Chih-chung Chang, and Chih-Jen Lin. 2003. “A Practical Guide to Support Vector Classification.”" - }, - { - "objectID": "gallery/2021-01-20-keras/2021-01-20-keras.html", - "href": "gallery/2021-01-20-keras/2021-01-20-keras.html", - "title": "Introduction to mlr3keras - Boston Housing", - "section": "", - "text": "The following article describes how to fit a Neural Network learner to the boston_housing dataset. This article is part of the mlr3keras vignette. For additional information or help on mlr3keras, please consult the mlr3keras website." - }, - { - "objectID": "gallery/2021-03-12-practical-tuning-series-tuning-and-parallel-processing/practical-tuning-series-tuning-and-parallel-processing.html", - "href": "gallery/2021-03-12-practical-tuning-series-tuning-and-parallel-processing/practical-tuning-series-tuning-and-parallel-processing.html", - "title": "Practical Tuning Series - Tuning and Parallel Processing", - "section": "", - "text": "Scope\nThis is the fourth part of the practical tuning series. The other parts can be found here:\n\nPart I - Tune a Support Vector Machine\nPart II - Tune a Preprocessing Pipeline\nPart III - Build an Automated Machine Learning System\n\nIn this post, we teach how to run various jobs in mlr3 in parallel. The goal is to map computational jobs (e.g. evaluation of one configuration) to a pool of workers (usually physical CPU cores, sometimes remote computational nodes) to reduce the run time needed for tuning.\n\n\nPrerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example. Additionally, make sure you have installed the packages future and future.apply.\n\nlibrary(mlr3verse)\n\nWe decrease the verbosity of the logger to keep the output clearly represented. The lgr package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning ).\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\n\n\nParallel Backend\nThe workers are specified by the parallel backend which orchestrates starting up, shutting down, and communication with the workers. On a single machine, multisession and multicore are common backends. The multisession backend spawns new background R processes. It is available on all platforms.\n\nfuture::plan(\"multisession\")\n\nThe multicore backend uses forked R processes which allows the workers to access R objects in a shared memory. This reduces the overhead since R objects are only copied in memory if they are modified. Unfortunately, forking processes is not supported on Windows and when running R from within RStudio.\n\nfuture::plan(\"multicore\")\n\nBoth backends support the workers argument that specifies the number of used cores.\nUse this code if your code should run with the multicore backend when possible.\n\nif (future::supportsMulticore()) {\n future::plan(future::multicore)\n} else {\n future::plan(future::multisession)\n}\n\n\n\nResampling\nThe resample() and benchmark() functions in mlr3 can be executed in parallel. The parallelization is triggered by simply declaring a plan via future::plan().\n\nfuture::plan(\"multisession\")\n\ntask = tsk(\"pima\")\nlearner = lrn(\"classif.rpart\") # classification tree\nresampling = rsmp(\"cv\", folds = 3)\n\nresample(task, learner, resampling)\n\n of 3 iterations\n* Task: pima\n* Learner: classif.rpart\n* Warnings: 0 in 0 iterations\n* Errors: 0 in 0 iterations\n\n\nThe 3-fold cross-validation gives us 3 jobs since each resampling iteration is executed in parallel.\nThe benchmark() function accepts a design of experiments as input where each experiment is defined as a combination of a task, a learner, and a resampling strategy. For each experiment, resampling is performed. The nested loop over experiments and resampling iterations is flattened so that all resampling iterations of all experiments can be executed in parallel.\n\nfuture::plan(\"multisession\")\n\ntasks = list(tsk(\"pima\"), tsk(\"iris\"))\nlearner = lrn(\"classif.rpart\")\nresampling = rsmp(\"cv\", folds = 3)\n\ngrid = benchmark_grid(tasks, learner, resampling)\n\nbenchmark(grid)\n\n of 6 rows with 2 resampling runs\n nr task_id learner_id resampling_id iters warnings errors\n 1 pima classif.rpart cv 3 0 0\n 2 iris classif.rpart cv 3 0 0\n\n\nThe 2 experiments and the 3-fold cross-validation result in 6 jobs which are executed in parallel.\n\n\nTuning\nThe mlr3tuning package internally calls benchmark() during tuning. If the tuner is capable of suggesting multiple configurations per iteration (such as random search, grid search, or hyperband), these configurations represent individual experiments, and the loop flattening of benchmark() is triggered. E.g., all resampling iterations of all hyperparameter configurations on a grid can be executed in parallel.\n\nfuture::plan(\"multisession\")\n\nlearner = lrn(\"classif.rpart\")\nlearner$param_set$values$cp = to_tune(0.001, 0.1)\nlearner$param_set$values$minsplit = to_tune(1, 10)\n\ninstance = tune(\n method = \"random_search\",\n task = tsk(\"pima\"),\n learner = learner,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 10,\n batch_size = 5 # random search suggests 5 configurations per batch\n)\n\nThe batch size of 5 and the 3-fold cross-validation gives us 15 jobs. This is done twice because of the limit of 10 evaluations in total.\n\n\nNested Resampling\nNested resampling results in two nested resampling loops. For this, an AutoTuner is passed to resample() or benchmark(). We can choose different parallelization backends for the inner and outer resampling loop, respectively. We just have to pass a list of backends.\n\n# Runs the outer loop in parallel and the inner loop sequentially\nfuture::plan(list(\"multisession\", \"sequential\"))\n\n# Runs the outer loop sequentially and the inner loop in parallel\nfuture::plan(list(\"sequential\", \"multisession\"))\n\nlearner = lrn(\"classif.rpart\")\nlearner$param_set$values$cp = to_tune(0.001, 0.1)\nlearner$param_set$values$minsplit = to_tune(1, 10)\n\nrr = tune_nested(\n method = \"random_search\",\n task = tsk(\"pima\"),\n learner = learner,\n inner_resampling = rsmp (\"cv\", folds = 3),\n outer_resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\"),\n term_evals = 10,\n batch_size = 5\n)\n\nWhile nesting real parallelization backends is often unintended and causes unnecessary overhead, it is useful in some distributed computing setups. It can be achieved with future by forcing a fixed number of workers for each loop.\n\n# Runs both loops in parallel\nfuture::plan(list(future::tweak(\"multisession\", workers = 2),\n future::tweak(\"multisession\", workers = 4)))\n\nThis example would run on 8 cores (= 2 * 4) on the local machine.\n\n\nResources\nThe mlr3book includes a chapters on parallelization. The mlr3cheatsheets contain frequently used commands and workflows of mlr3." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html", - "title": "Tuning a Complex Graph", - "section": "", - "text": "In this use case we show how to tune a rather complex graph consisting of different preprocessing steps and different learners where each preprocessing step and learner itself has parameters that can be tuned. You will learn the following:\nIdeally you already had a look at how to tune over multiple learners.\nFirst, we load the packages we will need:\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented. The lgr package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning )." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#data-and-task", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#data-and-task", - "title": "Tuning a Complex Graph", - "section": "Data and Task", - "text": "Data and Task\nWe are going to work with some gene expression data included as a supplement in the bst package. The data consists of 2308 gene profiles in 63 training and 20 test samples. The following data preprocessing steps are done analogously as in vignette(\"khan\", package = \"bst\"):\n\ndatafile = system.file(\"extdata\", \"supplemental_data\", package = \"bst\")\ndat0 = read.delim(datafile, header = TRUE, skip = 1)[, -(1:2)]\ndat0 = t(dat0)\ndat = data.frame(dat0[!(rownames(dat0) %in%\n c(\"TEST.9\", \"TEST.13\", \"TEST.5\", \"TEST.3\", \"TEST.11\")), ])\ndat$class = as.factor(\n c(substr(rownames(dat)[1:63], start = 1, stop = 2),\n c(\"NB\", \"RM\", \"NB\", \"EW\", \"RM\", \"BL\", \"EW\", \"RM\", \"EW\", \"EW\", \"EW\", \"RM\",\n \"BL\", \"RM\", \"NB\", \"NB\", \"NB\", \"NB\", \"BL\", \"EW\")\n )\n)\n\nWe then construct our training and test Task :\n\ntask = as_task_classif(dat, target = \"class\", id = \"SRBCT\")\ntask_train = task$clone(deep = TRUE)\ntask_train$filter(1:63)\ntask_test = task$clone(deep = TRUE)\ntask_test$filter(64:83)" - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#workflow", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#workflow", - "title": "Tuning a Complex Graph", - "section": "Workflow", - "text": "Workflow\nOur graph will start with log transforming the features, followed by scaling them. Then, either a PCA or ICA is applied to extract principal / independent components followed by fitting a LDA or a ranger random forest is fitted without any preprocessing (the log transformation and scaling should most likely affect the LDA more than the ranger random forest). Regarding the PCA and ICA, both the number of principal / independent components are tuning parameters. Regarding the LDA, we can further choose different methods for estimating the mean and variance and regarding the ranger, we want to tune the mtry and num.tree parameters. Note that the PCA-LDA combination has already been successfully applied in different cancer diagnostic contexts when the feature space is of high dimensionality (Morais and Lima 2018).\nTo allow for switching between the PCA / ICA-LDA and ranger we can either use branching or proxy pipelines, i.e., PipeOpBranch and PipeOpUnbranch or PipeOpProxy. We will first cover branching in detail and later show how the same can be done using PipeOpProxy." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#baseline", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#baseline", - "title": "Tuning a Complex Graph", - "section": "Baseline", - "text": "Baseline\nFirst, we have a look at the baseline classification accuracy of the LDA and ranger on the training task:\n\nbase = benchmark(benchmark_grid(\n task_train,\n learners = list(lrn(\"classif.lda\"), lrn(\"classif.ranger\")),\n resamplings = rsmp(\"cv\", folds = 3)))\n\nWarning in lda.default(x, grouping, ...): variables are collinear\n\nWarning in lda.default(x, grouping, ...): variables are collinear\n\nWarning in lda.default(x, grouping, ...): variables are collinear\n\nbase$aggregate(measures = msr(\"classif.acc\"))\n\n nr resample_result task_id learner_id resampling_id iters classif.acc\n1: 1 SRBCT classif.lda cv 3 0.6666667\n2: 2 SRBCT classif.ranger cv 3 0.9682540\n\n\nThe out-of-the-box ranger appears to already have good performance on the training task. Regarding the LDA, we do get a warning message that some features are colinear. This strongly suggests to reduce the dimensionality of the feature space. Let’s see if we can get some better performance, at least for the LDA." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#branching", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#branching", - "title": "Tuning a Complex Graph", - "section": "Branching", - "text": "Branching\nOur graph starts with log transforming the features (we explicitly use base 10 only for better interpretability when inspecting the model later), using PipeOpColApply, followed by scaling the features using PipeOpScale. Then, the first branch allows for switching between the PCA / ICA-LDA and ranger, and within PCA / ICA-LDA, the second branch allows for switching between PCA and ICA:\n\ngraph1 =\n po(\"colapply\", applicator = function(x) log(x, base = 10)) %>>%\n po(\"scale\") %>>%\n # pca / ica followed by lda vs. ranger\n po(\"branch\", id = \"branch_learner\", options = c(\"pca_ica_lda\", \"ranger\")) %>>%\n gunion(list(\n po(\"branch\", id = \"branch_preproc_lda\", options = c(\"pca\", \"ica\")) %>>%\n gunion(list(\n po(\"pca\"), po(\"ica\")\n )) %>>%\n po(\"unbranch\", id = \"unbranch_preproc_lda\") %>>%\n lrn(\"classif.lda\"),\n lrn(\"classif.ranger\")\n )) %>>%\n po(\"unbranch\", id = \"unbranch_learner\")\n\nNote that the names of the options within each branch are arbitrary, but ideally they describe what is happening. Therefore we go with \"pca_ica_lda\" / \"ranger” and \"pca\" / \"ica\". Finally, we also could have used the branch ppl to make branching easier (we will come back to this in the Proxy section). The graph looks like the following:\n\ngraph1$plot(html = TRUE)\n\n\n\n\n\nWe can inspect the parameters of the ParamSet of the graph to see which parameters can be set:\n\ngraph1$param_set$ids()\n\n [1] \"colapply.applicator\" \"colapply.affect_columns\" \n [3] \"scale.center\" \"scale.scale\" \n [5] \"scale.robust\" \"scale.affect_columns\" \n [7] \"branch_learner.selection\" \"branch_preproc_lda.selection\" \n [9] \"pca.center\" \"pca.scale.\" \n[11] \"pca.rank.\" \"pca.affect_columns\" \n[13] \"ica.n.comp\" \"ica.alg.typ\" \n[15] \"ica.fun\" \"ica.alpha\" \n[17] \"ica.method\" \"ica.row.norm\" \n[19] \"ica.maxit\" \"ica.tol\" \n[21] \"ica.verbose\" \"ica.w.init\" \n[23] \"ica.affect_columns\" \"classif.lda.dimen\" \n[25] \"classif.lda.method\" \"classif.lda.nu\" \n[27] \"classif.lda.predict.method\" \"classif.lda.predict.prior\" \n[29] \"classif.lda.prior\" \"classif.lda.tol\" \n[31] \"classif.ranger.alpha\" \"classif.ranger.always.split.variables\" \n[33] \"classif.ranger.class.weights\" \"classif.ranger.holdout\" \n[35] \"classif.ranger.importance\" \"classif.ranger.keep.inbag\" \n[37] \"classif.ranger.max.depth\" \"classif.ranger.min.node.size\" \n[39] \"classif.ranger.min.prop\" \"classif.ranger.minprop\" \n[41] \"classif.ranger.mtry\" \"classif.ranger.mtry.ratio\" \n[43] \"classif.ranger.num.random.splits\" \"classif.ranger.num.threads\" \n[45] \"classif.ranger.num.trees\" \"classif.ranger.oob.error\" \n[47] \"classif.ranger.regularization.factor\" \"classif.ranger.regularization.usedepth\" \n[49] \"classif.ranger.replace\" \"classif.ranger.respect.unordered.factors\" \n[51] \"classif.ranger.sample.fraction\" \"classif.ranger.save.memory\" \n[53] \"classif.ranger.scale.permutation.importance\" \"classif.ranger.se.method\" \n[55] \"classif.ranger.seed\" \"classif.ranger.split.select.weights\" \n[57] \"classif.ranger.splitrule\" \"classif.ranger.verbose\" \n[59] \"classif.ranger.write.forest\" \n\n\nThe id’s are prefixed by the respective PipeOp they belong to, e.g., pca.rank. refers to the rank. parameter of PipeOpPCA." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#search-space", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#search-space", - "title": "Tuning a Complex Graph", - "section": "Search Space", - "text": "Search Space\nOur graph either fits a LDA after applying PCA or ICA, or alternatively a ranger with no preprocessing. These two options each define selection parameters that we can tune. Moreover, within the respective PipeOp’s we want to tune the following parameters: pca.rank., ica.n.comp, classif.lda.method, classif.ranger.mtry, and classif.ranger.num.trees. The first two parameters are integers that in-principal could range from 1 to the number of features. However, for ICA, the upper bound must not exceed the number of observations and as we will later use 3-fold cross-validation as the resampling method for the tuning, we just set the upper bound to 30 (and do the same for PCA). Regarding the classif.lda.method we will only be interested in \"moment\" estimation vs. minimum volume ellipsoid covariance estimation (\"mve\"). Moreover, we set the lower bound of classif.ranger.mtry to 200 (which is around the number of features divided by 10) and the upper bound to 1000.\n\ntune_ps1 = ps(\n branch_learner.selection =\n p_fct(c(\"pca_ica_lda\", \"ranger\")),\n branch_preproc_lda.selection =\n p_fct(c(\"pca\", \"ica\"), depends = branch_learner.selection == \"pca_ica_lda\"),\n pca.rank. =\n p_int(1, 30, depends = branch_preproc_lda.selection == \"pca\"),\n ica.n.comp =\n p_int(1, 30, depends = branch_preproc_lda.selection == \"ica\"),\n classif.lda.method =\n p_fct(c(\"moment\", \"mve\"), depends = branch_preproc_lda.selection == \"ica\"),\n classif.ranger.mtry =\n p_int(200, 1000, depends = branch_learner.selection == \"ranger\"),\n classif.ranger.num.trees =\n p_int(500, 2000, depends = branch_learner.selection == \"ranger\"))\n\nThe parameter branch_learner.selection defines whether we go down the left (PCA / ICA followed by LDA) or the right branch (ranger). The parameter branch_preproc_lda.selection defines whether a PCA or ICA will be applied prior to the LDA. The other parameters directly belong to the ParamSet of the PCA / ICA / LDA / ranger. Note that it only makes sense to switch between PCA / ICA if the \"pca_ica_lda\" branch was selected beforehand. We have to specify this via the depends parameter.\nFinally, we also could have proceeded to tune the numeric parameters on a log scale. I.e., looking at pca.rank. the performance difference between rank 1 and 2 is probably much larger than between rank 29 and rank 30. The mlr3tuning Tutorial covers such transformations." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#tuning", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#tuning", - "title": "Tuning a Complex Graph", - "section": "Tuning", - "text": "Tuning\nWe can now tune the parameters of our graph as defined in the search space with respect to a measure. We will use the classification accuracy. As a resampling method we use 3-fold cross-validation. We will use the TerminatorNone (i.e., no early termination) for terminating the tuning because we will apply a grid search (we use a grid search because it gives nicely plottable and understandable results but if there were much more parameters, random search or more intelligent optimization methods would be preferred to a grid search:\n\ntune1 = TuningInstanceSingleCrit$new(\n task_train,\n learner = graph1,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.acc\"),\n search_space = tune_ps1,\n terminator = trm(\"none\")\n)\n\nWe then perform a grid search using a resolution of 4 for the numeric parameters. The grid being used will look like the following (note that the dependencies we specified above are handled automatically):\n\ngenerate_design_grid(tune_ps1, resolution = 4)\n\n\n\n\n\n\n\n\nWe trigger the tuning.\n\ntuner_gs = tnr(\"grid_search\", resolution = 4, batch_size = 10)\ntuner_gs$optimize(tune1)\n\n branch_learner.selection branch_preproc_lda.selection pca.rank. ica.n.comp classif.lda.method classif.ranger.mtry\n1: pca_ica_lda ica NA 10 mve NA\n classif.ranger.num.trees learner_param_vals x_domain classif.acc\n1: NA 0.984127\n\n\n\n\n\n\n\n\nNow, we can inspect the results ordered by the classification accuracy:\n\nas.data.table(tune1$archive)[order(classif.acc), ]\n\n\n\n\n\n\n\n\nWe achieve very good accuracy using ranger, more or less regardless how mtry and num.trees are set. However, the LDA also shows very good accuracy when combined with PCA or ICA retaining 30 components.\nFor now, we decide to use ranger with mtry set to 200 and num.trees set to 1000.\nSetting these parameters manually in our graph, then training on the training task and predicting on the test task yields an accuracy of:\n\ngraph1$param_set$values$branch_learner.selection = \"ranger\"\ngraph1$param_set$values$classif.ranger.mtry = 200\ngraph1$param_set$values$classif.ranger.num.trees = 1000\ngraph1$train(task_train)\n\n$unbranch_learner.output\nNULL\n\ngraph1$predict(task_test)[[1L]]$score(msr(\"classif.acc\"))\n\nclassif.acc \n 1 \n\n\nNote that we also could have wrapped our graph in a GraphLearner and proceeded to use this as a learner in an AutoTuner." - }, - { - "objectID": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#proxy", - "href": "gallery/2021-02-03-tuning-a-complex-graph/tuning-a-complex-graph.html#proxy", - "title": "Tuning a Complex Graph", - "section": "Proxy", - "text": "Proxy\nInstead of using branches to split our graph with respect to the learner and preprocessing options, we can also use PipeOpProxy. PipeOpProxy accepts a single content parameter that can contain any other PipeOp or Graph. This is extremely flexible in the sense that we do not have to specify our options during construction. However, the parameters of the contained PipeOp or Graph are no longer directly contained in the ParamSet of the resulting graph. Therefore, when tuning the graph, we do have to make use of a trafo function.\n\ngraph2 =\n po(\"colapply\", applicator = function(x) log(x, base = 10)) %>>%\n po(\"scale\") %>>%\n po(\"proxy\")\n\nThis graph now looks like the following:\n\ngraph2$plot(html = TRUE)\n\n\n\n\n\nAt first, this may look like a linear graph. However, as the content parameter of PipeOpProxy can be tuned and set to contain any other PipeOp or Graph, this will allow for a similar non-linear graph as when doing branching.\n\ngraph2$param_set$ids()\n\n[1] \"colapply.applicator\" \"colapply.affect_columns\" \"scale.center\" \"scale.scale\" \n[5] \"scale.robust\" \"scale.affect_columns\" \"proxy.content\" \n\n\nWe can tune the graph by using the same search space as before. However, here the trafo function is of central importance to actually set our options and parameters:\n\ntune_ps2 = tune_ps1$clone(deep = TRUE)\n\nThe trafo function does all the work, i.e., selecting either the PCA / ICA-LDA or ranger as the proxy.content as well as setting the parameters of the respective preprocessing PipeOps and Learners.\n\nproxy_options = list(\n pca_ica_lda =\n ppl(\"branch\", graphs = list(pca = po(\"pca\"), ica = po(\"ica\"))) %>>%\n lrn(\"classif.lda\"),\n ranger = lrn(\"classif.ranger\")\n)\n\nAbove, we made use of the branch ppl allowing us to easily construct a branching graph. Of course we also could have use another nested PipeOpProxy to specify the preprocessing options (\"pca\" vs. \"ica\") within proxy_options if for some reason we do not want to do branching at all. The trafo function below selects one of the proxy_options from above and sets the respective parameters for the PCA, ICA, LDA and ranger. Here, the argument x is a list which will contain sampled / selected parameters from our ParamSet (in our case, tune_ps2). The return value is a list only including the appropriate proxy.content parameter. In each tuning iteration, the proxy.content parameter of our graph will be set to this value.\n\ntune_ps2$trafo = function(x, param_set) {\n proxy.content = proxy_options[[x$branch_learner.selection]]\n if (x$branch_learner.selection == \"pca_ica_lda\") {\n # pca_ica_lda\n proxy.content$param_set$values$branch.selection = x$branch_preproc_lda.selection\n if (x$branch_preproc_lda.selection == \"pca\") {\n proxy.content$param_set$values$pca.rank. = x$pca.rank.\n } else {\n proxy.content$param_set$values$ica.n.comp = x$ica.n.comp\n }\n proxy.content$param_set$values$classif.lda.method = x$classif.lda.method\n } else {\n # ranger\n proxy.content$param_set$values$mtry = x$classif.ranger.mtry\n proxy.content$param_set$values$num.trees = x$classif.ranger.num.trees\n }\n list(proxy.content = proxy.content)\n}\n\nI.e., suppose that the following parameters will be selected from our ParamSet:\n\nx = list(\n branch_learner.selection = \"ranger\",\n classif.ranger.mtry = 200,\n classif.ranger.num.trees = 500)\n\nThe trafo function will then return:\n\ntune_ps2$trafo(x)\n\n$proxy.content\n\n* Model: -\n* Parameters: num.threads=1, mtry=200, num.trees=500\n* Packages: mlr3, mlr3learners, ranger\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, character, factor, ordered\n* Properties: hotstart_backward, importance, multiclass, oob_error, twoclass, weights\n\n\nTuning can be carried out analogously as done above:\n\ntune2 = TuningInstanceSingleCrit$new(\n task_train,\n learner = graph2,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.acc\"),\n search_space = tune_ps2,\n terminator = trm(\"none\")\n)\ntuner_gs$optimize(tune2)\n\n\nas.data.table(tune2$archive)[order(classif.acc), ]" - }, - { - "objectID": "gallery/2019-08-03-useR-mlr3pipelines/2019-08-03-useR-mlr3pipelines.html", - "href": "gallery/2019-08-03-useR-mlr3pipelines/2019-08-03-useR-mlr3pipelines.html", - "title": "useR! 2019 Presentation - mlr3pipelines", - "section": "", - "text": "Please note, that there have been slight changes in the syntax since the creation of this video." - }, - { - "objectID": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html", - "href": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html", - "title": "Comparison of Decision Boundaries of Classification Learners", - "section": "", - "text": "The visualization of decision boundaries helps to understand what the pros and cons of individual classification learners are. This posts demonstrates how to create such plots.\nWe load the mlr3 package.\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented." - }, - { - "objectID": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#artificial-data-sets", - "href": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#artificial-data-sets", - "title": "Comparison of Decision Boundaries of Classification Learners", - "section": "Artificial Data Sets", - "text": "Artificial Data Sets\nThe three artificial data sets are generated by task generators (implemented in mlr3):\n\nN = 200\ntasks = list(\n tgen(\"xor\")$generate(N),\n tgen(\"moons\")$generate(N),\n tgen(\"circle\")$generate(N)\n)\n\n\nXOR\nPoints are distributed on a 2-dimensional cube with corners \\((\\pm 1, \\pm 1)\\). Class is \"red\" if \\(x\\) and \\(y\\) have the same sign, and \"black\" otherwise.\n\nplot(tgen(\"xor\"))\n\n\n\n\n\n\n\n\n\n\nCircle\nTwo circles with same center but different radii. Points in the smaller circle are \"black\", points only in the larger circle are \"red\".\n\nplot(tgen(\"circle\"))\n\n\n\n\n\n\n\n\n\n\nMoons\nTwo interleaving half circles (“moons”).\n\nplot(tgen(\"moons\"))" - }, - { - "objectID": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#learners", - "href": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#learners", - "title": "Comparison of Decision Boundaries of Classification Learners", - "section": "Learners", - "text": "Learners\nWe consider the following learners:\n\nlibrary(\"mlr3learners\")\n\nlearners = list(\n # k-nearest neighbours classifier\n lrn(\"classif.kknn\", id = \"kkn\", predict_type = \"prob\", k = 3),\n\n # linear svm\n lrn(\"classif.svm\", id = \"lin. svm\", predict_type = \"prob\", kernel = \"linear\"),\n\n # radial-basis function svm\n lrn(\"classif.svm\", id = \"rbf svm\", predict_type = \"prob\", kernel = \"radial\",\n gamma = 2, cost = 1, type = \"C-classification\"),\n\n # naive bayes\n lrn(\"classif.naive_bayes\", id = \"naive bayes\", predict_type = \"prob\"),\n\n # single decision tree\n lrn(\"classif.rpart\", id = \"tree\", predict_type = \"prob\", cp = 0, maxdepth = 5),\n\n # random forest\n lrn(\"classif.ranger\", id = \"random forest\", predict_type = \"prob\")\n)\n\nThe hyperparameters are chosen in a way that the decision boundaries look “typical” for the respective classifier. Of course, with different hyperparameters, results may look very different." - }, - { - "objectID": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#fitting-the-models", - "href": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#fitting-the-models", - "title": "Comparison of Decision Boundaries of Classification Learners", - "section": "Fitting the Models", - "text": "Fitting the Models\nTo apply each learner on each task, we first build an exhaustive grid design of experiments with benchmark_grid() and then pass it to benchmark() to do the actual work. A simple holdout resampling is used here:\n\ndesign = benchmark_grid(\n tasks = tasks,\n learners = learners,\n resamplings = rsmp(\"holdout\")\n)\n\nbmr = benchmark(design, store_models = TRUE)\n\nA quick look into the performance values:\n\nperf = bmr$aggregate(msr(\"classif.acc\"))[, c(\"task_id\", \"learner_id\", \"classif.acc\")]\nperf\n task_id learner_id classif.acc\n1: xor_200 kkn 0.9402985 2: xor_200 lin. svm 0.5223881 3: xor_200 rbf svm 0.9701493 4: xor_200 naive bayes 0.4328358 5: xor_200 tree 0.9402985 6: xor_200 random forest 1.0000000 7: moons_200 kkn 1.0000000 8: moons_200 lin. svm 0.8805970 9: moons_200 rbf svm 1.0000000 10: moons_200 naive bayes 0.8955224 11: moons_200 tree 0.8955224 12: moons_200 random forest 0.9402985 13: circle_200 kkn 0.8805970 14: circle_200 lin. svm 0.4925373 15: circle_200 rbf svm 0.8955224 16: circle_200 naive bayes 0.7014925 17: circle_200 tree 0.7462687 18: circle_200 random forest 0.7910448\n\n\n\n\n\n \n \n task_id \n learner_id \n classif.acc \n \n \n\n \n xor_200 \n kkn \n 0.9402985 \n \n \n xor_200 \n lin. svm \n 0.5223881 \n \n \n xor_200 \n rbf svm \n 0.9701493 \n \n \n xor_200 \n naive bayes \n 0.4328358 \n \n \n xor_200 \n tree \n 0.9402985 \n \n \n xor_200 \n random forest \n 1.0000000 \n \n \n moons_200 \n kkn \n 1.0000000 \n \n \n moons_200 \n lin. svm \n 0.8805970 \n \n \n moons_200 \n rbf svm \n 1.0000000 \n \n \n moons_200 \n naive bayes \n 0.8955224 \n \n \n moons_200 \n tree \n 0.8955224 \n \n \n moons_200 \n random forest \n 0.9402985 \n \n \n circle_200 \n kkn \n 0.8805970 \n \n \n circle_200 \n lin. svm \n 0.4925373 \n \n \n circle_200 \n rbf svm \n 0.8955224 \n \n \n circle_200 \n naive bayes \n 0.7014925 \n \n \n circle_200 \n tree \n 0.7462687 \n \n \n circle_200 \n random forest \n 0.7910448" - }, - { - "objectID": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#plotting", - "href": "gallery/2020-08-14-comparison-of-decision-boundaries/comparison-of-decision-boundaries.html#plotting", - "title": "Comparison of Decision Boundaries of Classification Learners", - "section": "Plotting", - "text": "Plotting\nTo generate the plots, we iterate over the individual ResampleResult objects stored in the BenchmarkResult, and in each iteration we store the plot of the learner prediction generated by the mlr3viz package.\n\nlibrary(\"mlr3viz\")\n\nn = bmr$n_resample_results\nplots = vector(\"list\", n)\nfor (i in seq_len(n)) {\n rr = bmr$resample_result(i)\n plots[[i]] = autoplot(rr, type = \"prediction\")\n}\n\nWe now have a list of plots. Each one can be printed individually:\n\nprint(plots[[1]])\n\n\n\n\n\n\n\n\nNote that only observations from the test data is plotted as points.\nTo get a nice annotated overview, we arranged all plots together in a single PDF file. The number in the upper right is the respective accuracy on the test set.\nAs you can see, the decision boundaries look very different. Some are linear, others are parallel to the axis, and yet others are highly non-linear. The boundaries are partly very smooth with a slow transition of probabilities, others are very abrupt. All these properties are important during model selection, and should be considered for your problem at hand." - }, - { - "objectID": "gallery/2021-07-06-introduction-to-mlr3tuningspaces/introduction-to-mlr3tuningspaces.html", - "href": "gallery/2021-07-06-introduction-to-mlr3tuningspaces/introduction-to-mlr3tuningspaces.html", - "title": "Introduction to mlr3tuningspaces", - "section": "", - "text": "Scope\nThe package mlr3tuningspaces offers a selection of published search spaces for many popular machine learning algorithms. In this post, we show how to tune a mlr3 learners with these search spaces.\n\n\nPrerequisites\nThe packages mlr3verse and mlr3tuningspaces are required for this demonstration:\n\nlibrary(mlr3verse)\nlibrary(mlr3tuningspaces)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nIn the example, we use the pima indian diabetes data set which is used to predict whether or not a patient has diabetes. The patients are characterized by 8 numeric features, some of them have missing values.\n\n# retrieve the task from mlr3\ntask = tsk(\"pima\")\n\n# generate a quick textual overview using the skimr package\nskimr::skim(task$data())\n\n\nData summary\n\n\nName\ntask$data()\n\n\nNumber of rows\n768\n\n\nNumber of columns\n9\n\n\nKey\nNULL\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n1\n\n\nnumeric\n8\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\ndiabetes\n0\n1\nFALSE\n2\nneg: 500, pos: 268\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nage\n0\n1.00\n33.24\n11.76\n21.00\n24.00\n29.00\n41.00\n81.00\n▇▃▁▁▁\n\n\nglucose\n5\n0.99\n121.69\n30.54\n44.00\n99.00\n117.00\n141.00\n199.00\n▁▇▇▃▂\n\n\ninsulin\n374\n0.51\n155.55\n118.78\n14.00\n76.25\n125.00\n190.00\n846.00\n▇▂▁▁▁\n\n\nmass\n11\n0.99\n32.46\n6.92\n18.20\n27.50\n32.30\n36.60\n67.10\n▅▇▃▁▁\n\n\npedigree\n0\n1.00\n0.47\n0.33\n0.08\n0.24\n0.37\n0.63\n2.42\n▇▃▁▁▁\n\n\npregnant\n0\n1.00\n3.85\n3.37\n0.00\n1.00\n3.00\n6.00\n17.00\n▇▃▂▁▁\n\n\npressure\n35\n0.95\n72.41\n12.38\n24.00\n64.00\n72.00\n80.00\n122.00\n▁▃▇▂▁\n\n\ntriceps\n227\n0.70\n29.15\n10.48\n7.00\n22.00\n29.00\n36.00\n99.00\n▆▇▁▁▁\n\n\n\n\n\n\n\nTuning Search Space\nFor tuning, it is important to create a search space that defines the type and range of the hyperparameters. A learner stores all information about its hyperparameters in the slot $param_set. Usually, we have to chose a subset of hyperparameters we want to tune.\n\nlrn(\"classif.rpart\")$param_set\n\n\n id class lower upper nlevels default value\n 1: cp ParamDbl 0 1 Inf 0.01 \n 2: keep_model ParamLgl NA NA 2 FALSE \n 3: maxcompete ParamInt 0 Inf Inf 4 \n 4: maxdepth ParamInt 1 30 30 30 \n 5: maxsurrogate ParamInt 0 Inf Inf 5 \n 6: minbucket ParamInt 1 Inf Inf \n 7: minsplit ParamInt 1 Inf Inf 20 \n 8: surrogatestyle ParamInt 0 1 2 0 \n 9: usesurrogate ParamInt 0 2 3 2 \n10: xval ParamInt 0 Inf Inf 10 0\n\n\n\n\nPackage\nAt the heart of mlr3tuningspaces is the R6 class TuningSpace. It stores a list of TuneToken, helper functions and additional meta information. The list of TuneToken can be directly applied to the $values slot of a learner’s ParamSet. The search spaces are stored in the mlr_tuning_spaces dictionary.\n\nas.data.table(mlr_tuning_spaces)\n\n key label learner n_values\n 1: classif.glmnet.default Default GLM with Elastic Net Regularization Classification classif.glmnet 2\n 2: classif.glmnet.rbv2 RandomBot GLM with Elastic Net Regularization Classification classif.glmnet 2\n 3: classif.kknn.default Default k-Nearest-Neighbor Classification classif.kknn 3\n 4: classif.kknn.rbv2 RandomBot k-Nearest-Neighbor Classification classif.kknn 1\n 5: classif.ranger.default Default Ranger Classification classif.ranger 4\n 6: classif.ranger.rbv2 RandomBot Ranger Classification classif.ranger 8\n 7: classif.rpart.default Default Classification Tree classif.rpart 3\n 8: classif.rpart.rbv2 RandomBot Classification Tree classif.rpart 4\n 9: classif.svm.default Default Support Vector Machine Classification classif.svm 4\n10: classif.svm.rbv2 RandomBot Support Vector Machine Classification classif.svm 5\n11: classif.xgboost.default Default Extreme Gradient Boosting Classification classif.xgboost 8\n12: classif.xgboost.rbv2 RandomBot Extreme Gradient Boosting Classification classif.xgboost 13\n13: regr.glmnet.default Default GLM with Elastic Net Regularization Regression regr.glmnet 2\n14: regr.glmnet.rbv2 RandomBot GLM with Elastic Net Regularization Regression regr.glmnet 2\n15: regr.kknn.default Default k-Nearest-Neighbor Regression regr.kknn 3\n16: regr.kknn.rbv2 RandomBot k-Nearest-Neighbor Regression regr.kknn 1\n17: regr.ranger.default Default Ranger Regression regr.ranger 4\n18: regr.ranger.rbv2 RandomBot Ranger Regression regr.ranger 7\n19: regr.rpart.default Default Regression Tree regr.rpart 3\n20: regr.rpart.rbv2 RandomBot Regression Tree regr.rpart 4\n21: regr.svm.default Default Support Vector Machine Regression regr.svm 4\n22: regr.svm.rbv2 RandomBot Support Vector Machine Regression regr.svm 5\n23: regr.xgboost.default Default Extreme Gradient Boosting Regression regr.xgboost 8\n24: regr.xgboost.rbv2 RandomBot Extreme Gradient Boosting Regression regr.xgboost 13\n key label learner n_values\n\n\nWe can use the sugar function lts() to retrieve a TuningSpace.\n\ntuning_space_rpart = lts(\"classif.rpart.default\")\ntuning_space_rpart\n\n: Default Classification Tree\n id lower upper levels logscale\n1: minsplit 2e+00 128.0 TRUE\n2: minbucket 1e+00 64.0 TRUE\n3: cp 1e-04 0.1 TRUE\n\n\nThe $values slot contains the list of of TuneToken.\n\ntuning_space_rpart$values\n\n$minsplit\nTuning over:\nrange [2, 128] (log scale)\n\n\n$minbucket\nTuning over:\nrange [1, 64] (log scale)\n\n\n$cp\nTuning over:\nrange [1e-04, 0.1] (log scale)\n\n\nWe apply the search space and tune the learner.\n\nlearner = lrn(\"classif.rpart\")\n\nlearner$param_set$values = tuning_space_rpart$values\n\ninstance = tune(\n method = \"random_search\",\n task = tsk(\"pima\"),\n learner = learner,\n resampling = rsmp (\"holdout\"),\n measure = msr(\"classif.ce\"),\n term_evals = 10)\n\ninstance$result\n\n minsplit minbucket cp learner_param_vals x_domain classif.ce\n1: 1.377705 2.369973 -5.610915 0.2265625\n\n\nWe can also get the learner with search space already applied from the TuningSpace.\n\nlearner = tuning_space_rpart$get_learner()\nprint(learner$param_set)\n\n\n id class lower upper nlevels default value\n 1: cp ParamDbl 0 1 Inf 0.01 \n 2: keep_model ParamLgl NA NA 2 FALSE \n 3: maxcompete ParamInt 0 Inf Inf 4 \n 4: maxdepth ParamInt 1 30 30 30 \n 5: maxsurrogate ParamInt 0 Inf Inf 5 \n 6: minbucket ParamInt 1 Inf Inf \n 7: minsplit ParamInt 1 Inf Inf 20 \n 8: surrogatestyle ParamInt 0 1 2 0 \n 9: usesurrogate ParamInt 0 2 3 2 \n10: xval ParamInt 0 Inf Inf 10 0\n\n\nThis method also allows to set constant parameters.\n\nlearner = tuning_space_rpart$get_learner(maxdepth = 15)\nprint(learner$param_set)\n\n\n id class lower upper nlevels default value\n 1: cp ParamDbl 0 1 Inf 0.01 \n 2: keep_model ParamLgl NA NA 2 FALSE \n 3: maxcompete ParamInt 0 Inf Inf 4 \n 4: maxdepth ParamInt 1 30 30 30 15\n 5: maxsurrogate ParamInt 0 Inf Inf 5 \n 6: minbucket ParamInt 1 Inf Inf \n 7: minsplit ParamInt 1 Inf Inf 20 \n 8: surrogatestyle ParamInt 0 1 2 0 \n 9: usesurrogate ParamInt 0 2 3 2 \n10: xval ParamInt 0 Inf Inf 10 0\n\n\nThe lts() function sets the default search space directly to a learner.\n\nlearner = lts(lrn(\"classif.rpart\", maxdepth = 15))\nprint(learner$param_set)\n\n\n id class lower upper nlevels default value\n 1: cp ParamDbl 0 1 Inf 0.01 \n 2: keep_model ParamLgl NA NA 2 FALSE \n 3: maxcompete ParamInt 0 Inf Inf 4 \n 4: maxdepth ParamInt 1 30 30 30 15\n 5: maxsurrogate ParamInt 0 Inf Inf 5 \n 6: minbucket ParamInt 1 Inf Inf \n 7: minsplit ParamInt 1 Inf Inf 20 \n 8: surrogatestyle ParamInt 0 1 2 0 \n 9: usesurrogate ParamInt 0 2 3 2 \n10: xval ParamInt 0 Inf Inf 10 0" - }, - { - "objectID": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html", - "href": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html", - "title": "Imbalanced Data Handling with mlr3", - "section": "", - "text": "This use case compares different approaches to handle class imbalance for the Optical Recognition of Handwritten Digits (optdigits) binary classification data set using the mlr3 package. We mainly focus on undersampling the majority class, oversampling the minority class, and the SMOTE imbalance correction (Chawla et al. 2002) that enriches the minority class with synthetic data. The use case requires prior knowledge in basic ML concepts (issues imbalanced data, hyperparameter tuning, nested cross-validation). The R packages mlr3, mlr3pipelines and mlr3tuning will be used. You can find most of the content here also in the mlr3book explained in a more detailed way.\nThese steps are performed:\n\nRetrieve data sets from OpenML\nDefine imbalance correction pipeline Graphs (undersampling, oversampling and SMOTE) with mlr3pipelines\nAutotune the Graph together with a learner using mlr3tuning\nBenchmark the autotuned Graph and visualize the results using mlr3viz" - }, - { - "objectID": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html#visualize-benchmark-results", - "href": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html#visualize-benchmark-results", - "title": "Imbalanced Data Handling with mlr3", - "section": "Visualize benchmark results", - "text": "Visualize benchmark results\n\nbmr$aggregate(measure)\n\n nr resample_result task_id learner_id resampling_id iters classif.fbeta\n1: 1 optdigits undersample.ranger.tuned holdout 1 0.9516129\n2: 2 optdigits oversample.ranger.tuned holdout 1 0.9362881\n3: 3 optdigits smote.ranger.tuned holdout 1 0.9541779\n\n# one value per boxplot since we used holdout as outer resampling\nautoplot(bmr, measure = measure)" - }, - { - "objectID": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html#visualize-the-tuning-path", - "href": "gallery/2020-03-30-imbalanced-data/2020-03-30-imbalanced-data.html#visualize-the-tuning-path", - "title": "Imbalanced Data Handling with mlr3", - "section": "Visualize the tuning path", - "text": "Visualize the tuning path\nWith store_models = TRUE we allow the benchmark() function to store each single model that was computed during tuning. Therefore, we can plot the tuning path of the best learner from the subsampling iterations:\n\nlibrary(ggplot2)\nbmr_data_learners = as.data.table(bmr)$learner\nutune_path = bmr_data_learners[[1]]$model$tuning_instance$archive$data\nutune_gg = ggplot(utune_path, aes(x = undersample.ratio, y = classif.fbeta)) +\n geom_point(size = 3) +\n geom_line() + ylim(0.9, 1)\n\notune_path = bmr_data_learners[[2]]$model$tuning_instance$archive$data\notune_gg = ggplot(otune_path, aes(x = oversample.ratio, y = classif.fbeta)) +\n geom_point(size = 3) +\n geom_line() + ylim(0.9, 1)\n\nstune_path = bmr_data_learners[[3]]$model$tuning_instance$archive$data\nstune_gg = ggplot(stune_path, aes(\n x = smote.dup_size,\n y = classif.fbeta, col = factor(smote.K))) +\n geom_point(size = 3) +\n geom_line() + ylim(0.9, 1)\n\nlibrary(ggpubr)\nggarrange(utune_gg, otune_gg, stune_gg, common.legend = TRUE, nrow = 1)\n\n\n\n\n\n\n\n\nThe results show that oversampling the minority class (for simple oversampling as well as for SMOTE) and undersampling the majority class yield a better performance for this specific data set." - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html", - "title": "mlr3 Basics - German Credit", - "section": "", - "text": "This is the first part in a serial of tutorials. The other parts of this series can be found here:\n\nPart II - Tuning\nPart III - Pipelines\n\nWe will walk through this tutorial interactively. The text is kept short to be followed in real time." - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#prerequisites", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#prerequisites", - "title": "mlr3 Basics - German Credit", - "section": "Prerequisites", - "text": "Prerequisites\nEnsure all packages used in this tutorial are installed. This includes the mlr3verse package, as well as other packages for data handling, cleaning and visualization which we are going to use (data.table, ggplot2, rchallenge, and skimr).\nThen, load the main packages we are going to use:\n\nlibrary(\"mlr3verse\")\nlibrary(\"mlr3learners\")\nlibrary(\"mlr3tuning\")\nlibrary(\"data.table\")\nlibrary(\"ggplot2\")\n\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#machine-learning-use-case-german-credit-data", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#machine-learning-use-case-german-credit-data", - "title": "mlr3 Basics - German Credit", - "section": "Machine Learning Use Case: German Credit Data", - "text": "Machine Learning Use Case: German Credit Data\nThe German credit data was originally donated in 1994 by Prof. Dr. Hans Hoffman of the University of Hamburg. A description can be found at the UCI repository. The goal is to classify people by their credit risk (good or bad) using 20 personal, demographic and financial features:\n\n\n\n\n\n\n\nFeature Name\nDescription\n\n\n\n\nage\nage in years\n\n\namount\namount asked by applicant\n\n\ncredit_history\npast credit history of applicant at this bank\n\n\nduration\nduration of the credit in months\n\n\nemployment_duration\npresent employment since\n\n\nforeign_worker\nis applicant foreign worker?\n\n\nhousing\ntype of apartment rented, owned, for free / no payment\n\n\ninstallment_rate\ninstallment rate in percentage of disposable income\n\n\njob\ncurrent job information\n\n\nnumber_credits\nnumber of existing credits at this bank\n\n\nother_debtors\nother debtors/guarantors present?\n\n\nother_installment_plans\nother installment plans the applicant is paying\n\n\npeople_liable\nnumber of people being liable to provide maintenance\n\n\npersonal_status_sex\ncombination of sex and personal status of applicant\n\n\npresent_residence\npresent residence since\n\n\nproperty\nproperties that applicant has\n\n\npurpose\nreason customer is applying for a loan\n\n\nsavings\nsavings accounts/bonds at this bank\n\n\nstatus\nstatus/balance of checking account at this bank\n\n\ntelephone\nis there any telephone registered for this customer?\n\n\n\n\nImporting the Data\nThe dataset we are going to use is a transformed version of this German credit dataset, as provided by the rchallenge package (this transformed dataset was proposed by Ulrike Grömping, with factors instead of dummy variables and corrected features):\n\ndata(\"german\", package = \"rchallenge\")\n\nFirst, we’ll do a thorough investigation of the dataset.\n\n\nExploring the Data\nWe can get a quick overview of our dataset using R’s summary function:\n\ndim(german)\n\n[1] 1000 21\n\nstr(german)\n\n'data.frame': 1000 obs. of 21 variables:\n $ status : Factor w/ 4 levels \"no checking account\",..: 1 1 2 1 1 1 1 1 4 2 ...\n $ duration : int 18 9 12 12 12 10 8 6 18 24 ...\n $ credit_history : Factor w/ 5 levels \"delay in paying off in the past\",..: 5 5 3 5 5 5 5 5 5 3 ...\n $ purpose : Factor w/ 11 levels \"others\",\"car (new)\",..: 3 1 10 1 1 1 1 1 4 4 ...\n $ amount : int 1049 2799 841 2122 2171 2241 3398 1361 1098 3758 ...\n $ savings : Factor w/ 5 levels \"unknown/no savings account\",..: 1 1 2 1 1 1 1 1 1 3 ...\n $ employment_duration : Factor w/ 5 levels \"unemployed\",\"< 1 yr\",..: 2 3 4 3 3 2 4 2 1 1 ...\n $ installment_rate : Ord.factor w/ 4 levels \">= 35\"<\"25 <= ... < 35\"<..: 4 2 2 3 4 1 1 2 4 1 ...\n $ personal_status_sex : Factor w/ 4 levels \"male : divorced/separated\",..: 2 3 2 3 3 3 3 3 2 2 ...\n $ other_debtors : Factor w/ 3 levels \"none\",\"co-applicant\",..: 1 1 1 1 1 1 1 1 1 1 ...\n $ present_residence : Ord.factor w/ 4 levels \"< 1 yr\"<\"1 <= ... < 4 yrs\"<..: 4 2 4 2 4 3 4 4 4 4 ...\n $ property : Factor w/ 4 levels \"unknown / no property\",..: 2 1 1 1 2 1 1 1 3 4 ...\n $ age : int 21 36 23 39 38 48 39 40 65 23 ...\n $ other_installment_plans: Factor w/ 3 levels \"bank\",\"stores\",..: 3 3 3 3 1 3 3 3 3 3 ...\n $ housing : Factor w/ 3 levels \"for free\",\"rent\",..: 1 1 1 1 2 1 2 2 2 1 ...\n $ number_credits : Ord.factor w/ 4 levels \"1\"<\"2-3\"<\"4-5\"<..: 1 2 1 2 2 2 2 1 2 1 ...\n $ job : Factor w/ 4 levels \"unemployed/unskilled - non-resident\",..: 3 3 2 2 2 2 2 2 1 1 ...\n $ people_liable : Factor w/ 2 levels \"3 or more\",\"0 to 2\": 2 1 2 1 2 1 2 1 2 2 ...\n $ telephone : Factor w/ 2 levels \"no\",\"yes (under customer name)\": 1 1 1 1 1 1 1 1 1 1 ...\n $ foreign_worker : Factor w/ 2 levels \"yes\",\"no\": 2 2 2 1 1 1 1 1 2 2 ...\n $ credit_risk : Factor w/ 2 levels \"bad\",\"good\": 2 2 2 2 2 2 2 2 2 2 ...\n\n\nOur dataset has 1000 observations and 21 columns. The variable we want to predict is credit_risk (either good or bad), i.e., we aim to classify people by their credit risk.\nWe also recommend the skimr package as it creates very well readable and understandable overviews:\n\nskimr::skim(german)\n\n\nData summary\n\n\nName\ngerman\n\n\nNumber of rows\n1000\n\n\nNumber of columns\n21\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n18\n\n\nnumeric\n3\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\nstatus\n0\n1\nFALSE\n4\n…: 394, no : 274, …: 269, 0<=: 63\n\n\ncredit_history\n0\n1\nFALSE\n5\nno : 530, all: 293, exi: 88, cri: 49\n\n\npurpose\n0\n1\nFALSE\n10\nfur: 280, oth: 234, car: 181, car: 103\n\n\nsavings\n0\n1\nFALSE\n5\nunk: 603, …: 183, …: 103, 100: 63\n\n\nemployment_duration\n0\n1\nFALSE\n5\n1 <: 339, >= : 253, 4 <: 174, < 1: 172\n\n\ninstallment_rate\n0\n1\nTRUE\n4\n< 2: 476, 25 : 231, 20 : 157, >= : 136\n\n\npersonal_status_sex\n0\n1\nFALSE\n4\nmal: 548, fem: 310, fem: 92, mal: 50\n\n\nother_debtors\n0\n1\nFALSE\n3\nnon: 907, gua: 52, co-: 41\n\n\npresent_residence\n0\n1\nTRUE\n4\n>= : 413, 1 <: 308, 4 <: 149, < 1: 130\n\n\nproperty\n0\n1\nFALSE\n4\nbui: 332, unk: 282, car: 232, rea: 154\n\n\nother_installment_plans\n0\n1\nFALSE\n3\nnon: 814, ban: 139, sto: 47\n\n\nhousing\n0\n1\nFALSE\n3\nren: 714, for: 179, own: 107\n\n\nnumber_credits\n0\n1\nTRUE\n4\n1: 633, 2-3: 333, 4-5: 28, >= : 6\n\n\njob\n0\n1\nFALSE\n4\nski: 630, uns: 200, man: 148, une: 22\n\n\npeople_liable\n0\n1\nFALSE\n2\n0 t: 845, 3 o: 155\n\n\ntelephone\n0\n1\nFALSE\n2\nno: 596, yes: 404\n\n\nforeign_worker\n0\n1\nFALSE\n2\nno: 963, yes: 37\n\n\ncredit_risk\n0\n1\nFALSE\n2\ngoo: 700, bad: 300\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nduration\n0\n1\n20.90\n12.06\n4\n12.0\n18.0\n24.00\n72\n▇▇▂▁▁\n\n\namount\n0\n1\n3271.25\n2822.75\n250\n1365.5\n2319.5\n3972.25\n18424\n▇▂▁▁▁\n\n\nage\n0\n1\n35.54\n11.35\n19\n27.0\n33.0\n42.00\n75\n▇▆▃▁▁\n\n\n\n\n\nDuring an exploratory analysis meaningful discoveries could be:\n\nSkewed distributions\nMissing values\nEmpty / rare factor variables\n\nAn explanatory analysis is crucial to get a feeling for your data. On the other hand the data can be validated this way. Non-plausible data can be investigated or outliers can be removed.\nAfter feeling confident with the data, we want to do modeling now." - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#modeling", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#modeling", - "title": "mlr3 Basics - German Credit", - "section": "Modeling", - "text": "Modeling\nConsidering how we are going to tackle the problem of classifying the credit risk relates closely to what mlr3 entities we will use.\nThe typical questions that arise when building a machine learning workflow are:\n\nWhat is the problem we are trying to solve?\nWhat are appropriate learning algorithms?\nHow do we evaluate “good” performance?\n\nMore systematically in mlr3 they can be expressed via five components:\n\nThe Task definition.\nThe Learner definition.\nThe training.\nThe prediction.\nThe evaluation via one or multiple Measures.\n\n\nTask Definition\nFirst, we are interested in the target which we want to model. Most supervised machine learning problems are regression or classification problems. However, note that other problems include unsupervised learning or time-to-event data (covered in mlr3proba).\nWithin mlr3, to distinguish between these problems, we define Tasks. If we want to solve a classification problem, we define a classification task – TaskClassif. For a regression problem, we define a regression task – TaskRegr.\nIn our case it is clearly our objective to model or predict the binary factor variable credit_risk. Thus, we define a TaskClassif:\n\ntask = as_task_classif(german, id = \"GermanCredit\", target = \"credit_risk\")\n\nNote that the German credit data is also given as an example task which ships with the mlr3 package. Thus, you actually don’t need to construct it yourself, just call tsk(\"german_credit\") to retrieve the object from the dictionary mlr_tasks.\n\n\nLearner Definition\nAfter having decided what should be modeled, we need to decide on how. This means we need to decide which learning algorithms, or Learners are appropriate. Using prior knowledge (e.g. knowing that it is a classification task or assuming that the classes are linearly separable) one ends up with one or more suitable Learners.\nMany learners can be obtained via the mlr3learners package. Additionally, many learners are provided via the mlr3extralearners package, from GitHub. These two resources combined account for a large fraction of standard learning algorithms. As mlr3 usually only wraps learners from packages, it is even easy to create a formal Learner by yourself. You may find the section about extending mlr3 in the mlr3book very helpful. If you happen to write your own Learner in mlr3, we would be happy if you share it with the mlr3 community.\nAll available Learners (i.e. all which you have installed from mlr3, mlr3learners, mlr3extralearners, or self-written ones) are registered in the dictionary mlr_learners:\n\nmlr_learners\n\n with 130 stored values\nKeys: classif.AdaBoostM1, classif.bart, classif.C50, classif.catboost, classif.cforest, classif.ctree,\n classif.cv_glmnet, classif.debug, classif.earth, classif.featureless, classif.fnn, classif.gam,\n classif.gamboost, classif.gausspr, classif.gbm, classif.glmboost, classif.glmnet, classif.IBk,\n classif.J48, classif.JRip, classif.kknn, classif.ksvm, classif.lda, classif.liblinear, classif.lightgbm,\n classif.LMT, classif.log_reg, classif.lssvm, classif.mob, classif.multinom, classif.naive_bayes,\n classif.nnet, classif.OneR, classif.PART, classif.qda, classif.randomForest, classif.ranger,\n classif.rfsrc, classif.rpart, classif.svm, classif.xgboost, clust.agnes, clust.ap, clust.cmeans,\n clust.cobweb, clust.dbscan, clust.diana, clust.em, clust.fanny, clust.featureless, clust.ff,\n clust.hclust, clust.kkmeans, clust.kmeans, clust.MBatchKMeans, clust.meanshift, clust.pam,\n clust.SimpleKMeans, clust.xmeans, dens.kde_ks, dens.locfit, dens.logspline, dens.mixed, dens.nonpar,\n dens.pen, dens.plug, dens.spline, regr.bart, regr.catboost, regr.cforest, regr.ctree, regr.cubist,\n regr.cv_glmnet, regr.debug, regr.earth, regr.featureless, regr.fnn, regr.gam, regr.gamboost,\n regr.gausspr, regr.gbm, regr.glm, regr.glmboost, regr.glmnet, regr.IBk, regr.kknn, regr.km, regr.ksvm,\n regr.liblinear, regr.lightgbm, regr.lm, regr.lmer, regr.M5Rules, regr.mars, regr.mob, regr.nnet,\n regr.randomForest, regr.ranger, regr.rfsrc, regr.rpart, regr.rvm, regr.svm, regr.xgboost, surv.akritas,\n surv.blackboost, surv.cforest, surv.coxboost, surv.coxtime, surv.ctree, surv.cv_coxboost, surv.cv_glmnet,\n surv.deephit, surv.deepsurv, surv.dnnsurv, surv.flexible, surv.gamboost, surv.gbm, surv.glmboost,\n surv.glmnet, surv.loghaz, surv.mboost, surv.nelson, surv.obliqueRSF, surv.parametric, surv.pchazard,\n surv.penalized, surv.ranger, surv.rfsrc, surv.svm, surv.xgboost\n\n\nFor our problem, a suitable learner could be one of the following: Logistic regression, CART, random forest (or many more).\nA learner can be initialized with the lrn() function and the name of the learner, e.g., lrn(\"classif.xxx\"). Use ?mlr_learners_xxx to open the help page of a learner named xxx.\nFor example, a logistic regression can be initialized in the following manner (logistic regression uses R’s glm() function and is provided by the mlr3learners package):\n\nlibrary(\"mlr3learners\")\nlearner_logreg = lrn(\"classif.log_reg\")\nprint(learner_logreg)\n\n\n* Model: -\n* Parameters: list()\n* Packages: mlr3, mlr3learners, stats\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, character, factor, ordered\n* Properties: loglik, twoclass\n\n\n\n\nTraining\nTraining is the procedure, where a model is fitted on the (training) data.\n\nLogistic Regression\nWe start with the example of the logistic regression. However, you will immediately see that the procedure generalizes to any learner very easily.\nAn initialized learner can be trained on data using $train():\n\nlearner_logreg$train(task)\n\nTypically, in machine learning, one does not use the full data which is available but a subset, the so-called training data.\nTo efficiently perform a split of the data one could do the following:\n\ntrain_set = sample(task$row_ids, 0.8 * task$nrow)\ntest_set = setdiff(task$row_ids, train_set)\n\n80 percent of the data is used for training. The remaining 20 percent are used for evaluation at a subsequent later point in time. train_set is an integer vector referring to the selected rows of the original dataset:\n\nhead(train_set)\n\n[1] 70 996 711 924 29 968\n\n\nIn mlr3 the training with a subset of the data can be declared by the additional argument row_ids = train_set:\n\nlearner_logreg$train(task, row_ids = train_set)\n\nThe fitted model can be accessed via:\n\nlearner_logreg$model\n\n\nCall: stats::glm(formula = task$formula(), family = \"binomial\", data = data, \n model = FALSE)\n\nCoefficients:\n (Intercept) age \n 4.405e-01 -6.299e-03 \n amount credit_historycritical account/other credits elsewhere \n 1.272e-04 8.369e-01 \ncredit_historyno credits taken/all credits paid back duly credit_historyexisting credits paid back duly till now \n -3.127e-01 -5.415e-01 \n credit_historyall credits at this bank paid back duly duration \n -1.015e+00 2.747e-02 \n employment_duration< 1 yr employment_duration1 <= ... < 4 yrs \n 3.424e-01 -1.080e-01 \n employment_duration4 <= ... < 7 yrs employment_duration>= 7 yrs \n -5.131e-01 -4.189e-02 \n foreign_workerno housingrent \n 1.303e+00 -5.993e-01 \n housingown installment_rate.L \n -6.745e-01 8.898e-01 \n installment_rate.Q installment_rate.C \n 4.186e-02 -6.346e-02 \n jobunskilled - resident jobskilled employee/official \n 8.655e-02 1.878e-01 \n jobmanager/self-empl./highly qualif. employee number_credits.L \n -6.201e-02 1.347e-01 \n number_credits.Q number_credits.C \n -2.025e-02 3.681e-01 \n other_debtorsco-applicant other_debtorsguarantor \n -3.834e-01 -9.652e-01 \n other_installment_plansstores other_installment_plansnone \n 2.218e-01 -2.182e-01 \n people_liable0 to 2 personal_status_sexfemale : non-single or male : single \n -2.579e-01 -5.771e-01 \n personal_status_sexmale : married/widowed personal_status_sexfemale : single \n -9.474e-01 -1.323e-01 \n present_residence.L present_residence.Q \n 2.759e-01 -4.435e-01 \n present_residence.C propertycar or other \n 3.448e-01 1.996e-01 \n propertybuilding soc. savings agr./life insurance propertyreal estate \n -2.296e-02 3.712e-01 \n purposecar (new) purposecar (used) \n -1.555e+00 -7.028e-01 \n purposefurniture/equipment purposeradio/television \n -9.382e-01 -8.289e-01 \n purposedomestic appliances purposerepairs \n -3.343e-01 1.444e-01 \n purposevacation purposeretraining \n -1.532e+01 -6.120e-01 \n purposebusiness savings... < 100 DM \n -1.315e+00 -4.635e-01 \n savings100 <= ... < 500 DM savings500 <= ... < 1000 DM \n -8.924e-01 -1.442e+00 \n savings... >= 1000 DM status... < 0 DM \n -8.856e-01 -4.287e-01 \n status0<= ... < 200 DM status... >= 200 DM / salary for at least 1 year \n -1.027e+00 -1.695e+00 \n telephoneyes (under customer name) \n -2.913e-01 \n\nDegrees of Freedom: 799 Total (i.e. Null); 745 Residual\nNull Deviance: 972.2 \nResidual Deviance: 712.5 AIC: 822.5\n\n\nThe stored object is a normal glm object and all its S3 methods work as expected:\n\nclass(learner_logreg$model)\n\n[1] \"glm\" \"lm\" \n\nsummary(learner_logreg$model)\n\n\nCall:\nstats::glm(formula = task$formula(), family = \"binomial\", data = data, \n model = FALSE)\n\nDeviance Residuals: \n Min 1Q Median 3Q Max \n-2.0212 -0.6904 -0.3731 0.6641 2.8750 \n\nCoefficients:\n Estimate Std. Error z value Pr(>|z|) \n(Intercept) 4.405e-01 1.371e+00 0.321 0.748029 \nage -6.299e-03 1.052e-02 -0.599 0.549207 \namount 1.272e-04 5.007e-05 2.541 0.011068 * \ncredit_historycritical account/other credits elsewhere 8.369e-01 6.468e-01 1.294 0.195672 \ncredit_historyno credits taken/all credits paid back duly -3.127e-01 4.943e-01 -0.633 0.527039 \ncredit_historyexisting credits paid back duly till now -5.415e-01 5.216e-01 -1.038 0.299173 \ncredit_historyall credits at this bank paid back duly -1.015e+00 4.925e-01 -2.061 0.039304 * \nduration 2.747e-02 1.048e-02 2.620 0.008790 ** \nemployment_duration< 1 yr 3.424e-01 4.885e-01 0.701 0.483327 \nemployment_duration1 <= ... < 4 yrs -1.080e-01 4.645e-01 -0.233 0.816086 \nemployment_duration4 <= ... < 7 yrs -5.131e-01 5.069e-01 -1.012 0.311420 \nemployment_duration>= 7 yrs -4.189e-02 4.669e-01 -0.090 0.928512 \nforeign_workerno 1.303e+00 7.188e-01 1.813 0.069827 . \nhousingrent -5.993e-01 2.642e-01 -2.268 0.023319 * \nhousingown -6.745e-01 5.372e-01 -1.256 0.209269 \ninstallment_rate.L 8.898e-01 2.536e-01 3.509 0.000450 ***\ninstallment_rate.Q 4.186e-02 2.275e-01 0.184 0.854006 \ninstallment_rate.C -6.346e-02 2.289e-01 -0.277 0.781628 \njobunskilled - resident 8.655e-02 8.149e-01 0.106 0.915418 \njobskilled employee/official 1.878e-01 7.891e-01 0.238 0.811888 \njobmanager/self-empl./highly qualif. employee -6.201e-02 8.049e-01 -0.077 0.938591 \nnumber_credits.L 1.347e-01 7.298e-01 0.185 0.853553 \nnumber_credits.Q -2.025e-02 6.363e-01 -0.032 0.974607 \nnumber_credits.C 3.681e-01 5.250e-01 0.701 0.483193 \nother_debtorsco-applicant -3.834e-01 5.374e-01 -0.714 0.475496 \nother_debtorsguarantor -9.652e-01 4.682e-01 -2.062 0.039234 * \nother_installment_plansstores 2.218e-01 4.587e-01 0.483 0.628804 \nother_installment_plansnone -2.182e-01 2.874e-01 -0.759 0.447727 \npeople_liable0 to 2 -2.579e-01 2.832e-01 -0.911 0.362361 \npersonal_status_sexfemale : non-single or male : single -5.771e-01 4.253e-01 -1.357 0.174819 \npersonal_status_sexmale : married/widowed -9.474e-01 4.119e-01 -2.300 0.021437 * \npersonal_status_sexfemale : single -1.323e-01 4.991e-01 -0.265 0.790972 \npresent_residence.L 2.759e-01 2.435e-01 1.133 0.257121 \npresent_residence.Q -4.435e-01 2.279e-01 -1.946 0.051649 . \npresent_residence.C 3.448e-01 2.330e-01 1.480 0.138929 \npropertycar or other 1.996e-01 2.856e-01 0.699 0.484629 \npropertybuilding soc. savings agr./life insurance -2.296e-02 2.682e-01 -0.086 0.931793 \npropertyreal estate 3.712e-01 4.712e-01 0.788 0.430825 \npurposecar (new) -1.555e+00 4.085e-01 -3.806 0.000141 ***\npurposecar (used) -7.028e-01 2.968e-01 -2.368 0.017885 * \npurposefurniture/equipment -9.382e-01 2.824e-01 -3.323 0.000891 ***\npurposeradio/television -8.289e-01 8.592e-01 -0.965 0.334700 \npurposedomestic appliances -3.343e-01 6.060e-01 -0.552 0.581168 \npurposerepairs 1.444e-01 4.609e-01 0.313 0.754141 \npurposevacation -1.532e+01 4.378e+02 -0.035 0.972083 \npurposeretraining -6.120e-01 3.706e-01 -1.651 0.098644 . \npurposebusiness -1.315e+00 9.904e-01 -1.328 0.184158 \nsavings... < 100 DM -4.635e-01 3.326e-01 -1.394 0.163448 \nsavings100 <= ... < 500 DM -8.924e-01 5.165e-01 -1.728 0.084066 . \nsavings500 <= ... < 1000 DM -1.442e+00 6.077e-01 -2.373 0.017651 * \nsavings... >= 1000 DM -8.856e-01 2.892e-01 -3.062 0.002199 ** \nstatus... < 0 DM -4.287e-01 2.520e-01 -1.701 0.088851 . \nstatus0<= ... < 200 DM -1.027e+00 4.346e-01 -2.362 0.018180 * \nstatus... >= 200 DM / salary for at least 1 year -1.695e+00 2.602e-01 -6.516 7.24e-11 ***\ntelephoneyes (under customer name) -2.913e-01 2.252e-01 -1.293 0.195868 \n---\nSignif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n\n(Dispersion parameter for binomial family taken to be 1)\n\n Null deviance: 972.25 on 799 degrees of freedom\nResidual deviance: 712.46 on 745 degrees of freedom\nAIC: 822.46\n\nNumber of Fisher Scoring iterations: 14\n\n\n\n\nRandom Forest\nJust like the logistic regression, we could train a random forest instead. We use the fast implementation from the ranger package. For this, we first need to define the learner and then actually train it.\nWe now additionally supply the importance argument (importance = \"permutation\"). Doing so, we override the default and let the learner do feature importance determination based on permutation feature importance:\n\nlearner_rf = lrn(\"classif.ranger\", importance = \"permutation\")\nlearner_rf$train(task, row_ids = train_set)\n\nWe can access the importance values using $importance():\n\nlearner_rf$importance()\n\n status duration amount credit_history savings \n 3.687484e-02 2.108981e-02 1.098572e-02 8.972925e-03 7.115591e-03 \n installment_rate age purpose employment_duration other_debtors \n 6.559872e-03 6.259569e-03 5.172414e-03 4.929074e-03 2.453441e-03 \n present_residence housing property other_installment_plans number_credits \n 2.227919e-03 1.796496e-03 1.773314e-03 1.558226e-03 1.552618e-03 \n job telephone personal_status_sex people_liable foreign_worker \n 1.506459e-03 1.266944e-03 1.007241e-03 -7.151404e-05 -1.230687e-04 \n\n\nIn order to obtain a plot for the importance values, we convert the importance to a data.table and then process it with ggplot2:\n\nimportance = as.data.table(learner_rf$importance(), keep.rownames = TRUE)\ncolnames(importance) = c(\"Feature\", \"Importance\")\nggplot(importance, aes(x = reorder(Feature, Importance), y = Importance)) +\n geom_col() + coord_flip() + xlab(\"\")\n\n\n\n\n\n\n\n\n\n\n\nPrediction\nLet’s see what the models predict.\nAfter training a model, the model can be used for prediction. Usually, prediction is the main purpose of machine learning models.\nIn our case, the model can be used to classify new credit applicants w.r.t. their associated credit risk (good vs. bad) on the basis of the features. Typically, machine learning models predict numeric values. In the regression case this is very natural. For classification, most models predict scores or probabilities. Based on these values, one can derive class predictions.\n\nPredict Classes\nFirst, we directly predict classes:\n\nprediction_logreg = learner_logreg$predict(task, row_ids = test_set)\nprediction_rf = learner_rf$predict(task, row_ids = test_set)\n\n\nprediction_logreg\n\n for 200 observations:\n row_ids truth response\n 20 good good\n 22 good good\n 25 good good\n--- \n 994 bad good\n 997 bad good\n 999 bad good\n\n\n\nprediction_rf\n\n for 200 observations:\n row_ids truth response\n 20 good good\n 22 good good\n 25 good good\n--- \n 994 bad good\n 997 bad bad\n 999 bad good\n\n\nThe $predict() method returns a Prediction object. It can be converted to a data.table if one wants to use it downstream.\nWe can also display the prediction results aggregated in a confusion matrix:\n\nprediction_logreg$confusion\n\n truth\nresponse bad good\n bad 27 14\n good 36 123\n\nprediction_rf$confusion\n\n truth\nresponse bad good\n bad 27 12\n good 36 125\n\n\n\n\nPredict Probabilities\nMost learners may not only predict a class variable (“response”), but also their degree of “belief” / “uncertainty” in a given response. Typically, we achieve this by setting the $predict_type slot of a Learner to \"prob\". Sometimes this needs to be done before the learner is trained. Alternatively, we can directly create the learner with this option: lrn(\"classif.log_reg\", predict_type = \"prob\").\n\nlearner_logreg$predict_type = \"prob\"\n\n\nlearner_logreg$predict(task, row_ids = test_set)\n\n for 200 observations:\n row_ids truth response prob.bad prob.good\n 20 good good 0.10625042 0.8937496\n 22 good good 0.33581567 0.6641843\n 25 good good 0.21620976 0.7837902\n--- \n 994 bad good 0.45300711 0.5469929\n 997 bad good 0.44431153 0.5556885\n 999 bad good 0.05065001 0.9493500\n\n\nNote that sometimes one needs to be cautious when dealing with the probability interpretation of the predictions.\n\n\n\nPerformance Evaluation\nTo measure the performance of a learner on new unseen data, we usually mimic the scenario of unseen data by splitting up the data into training and test set. The training set is used for training the learner, and the test set is only used for predicting and evaluating the performance of the trained learner. Numerous resampling methods (cross-validation, bootstrap) repeat the splitting process in different ways.\nWithin mlr3, we need to specify the resampling strategy using the rsmp() function:\n\nresampling = rsmp(\"holdout\", ratio = 2/3)\nprint(resampling)\n\n: Holdout\n* Iterations: 1\n* Instantiated: FALSE\n* Parameters: ratio=0.6667\n\n\nHere, we use “holdout”, a simple train-test split (with just one iteration). We use the resample() function to undertake the resampling calculation:\n\nres = resample(task, learner = learner_logreg, resampling = resampling)\nres\n\n of 1 iterations\n* Task: GermanCredit\n* Learner: classif.log_reg\n* Warnings: 0 in 0 iterations\n* Errors: 0 in 0 iterations\n\n\nThe default score of the measure is included in the $aggregate() slot:\n\nres$aggregate()\n\nclassif.ce \n 0.2162162 \n\n\nThe default measure in this scenario is the classification error. Lower is better.\nWe can easily run different resampling strategies, e.g. repeated holdout (\"subsampling\"), or cross validation. Most methods perform repeated train/predict cycles on different data subsets and aggregate the result (usually as the mean). Doing this manually would require us to write loops. mlr3 does the job for us:\n\nresampling = rsmp(\"subsampling\", repeats = 10)\nrr = resample(task, learner = learner_logreg, resampling = resampling)\nrr$aggregate()\n\nclassif.ce \n 0.2513514 \n\n\nInstead, we could also run cross-validation:\n\nresampling = resampling = rsmp(\"cv\", folds = 10)\nrr = resample(task, learner = learner_logreg, resampling = resampling)\nrr$aggregate()\n\nclassif.ce \n 0.256 \n\n\nmlr3 features scores for many more measures. Here, we apply mlr_measures_classif.fpr for the false positive rate, and mlr_measures_classif.fnr for the false negative rate. Multiple measures can be provided as a list of measures (which can directly be constructed via msrs():\n\n# false positive rate\nrr$aggregate(msr(\"classif.fpr\"))\n\nclassif.fpr \n 0.1427762 \n\n# false positive rate and false negative\nmeasures = msrs(c(\"classif.fpr\", \"classif.fnr\"))\nrr$aggregate(measures)\n\nclassif.fpr classif.fnr \n 0.1427762 0.5218063 \n\n\nThere are a few more resampling methods, and quite a few more measures (implemented in mlr3measures). They are automatically registered in the respective dictionaries:\n\nmlr_resamplings\n\n with 9 stored values\nKeys: bootstrap, custom, custom_cv, cv, holdout, insample, loo, repeated_cv, subsampling\n\nmlr_measures\n\n with 67 stored values\nKeys: aic, bic, classif.acc, classif.auc, classif.bacc, classif.bbrier, classif.ce, classif.costs,\n classif.dor, classif.fbeta, classif.fdr, classif.fn, classif.fnr, classif.fomr, classif.fp, classif.fpr,\n classif.logloss, classif.mauc_au1p, classif.mauc_au1u, classif.mauc_aunp, classif.mauc_aunu,\n classif.mbrier, classif.mcc, classif.npv, classif.ppv, classif.prauc, classif.precision, classif.recall,\n classif.sensitivity, classif.specificity, classif.tn, classif.tnr, classif.tp, classif.tpr, clust.ch,\n clust.db, clust.dunn, clust.silhouette, clust.wss, debug, oob_error, regr.bias, regr.ktau, regr.mae,\n regr.mape, regr.maxae, regr.medae, regr.medse, regr.mse, regr.msle, regr.pbias, regr.rae, regr.rmse,\n regr.rmsle, regr.rrse, regr.rse, regr.rsq, regr.sae, regr.smape, regr.srho, regr.sse, selected_features,\n sim.jaccard, sim.phi, time_both, time_predict, time_train\n\n\nTo get help on a resampling method, use ?mlr_resamplings_xxx, for a measure do ?mlr_measures_xxx. You can also browse the mlr3 reference online.\nNote that some measures, for example AUC, require the prediction of probabilities.\n\n\nPerformance Comparison and Benchmarks\nWe could compare Learners by evaluating resample() for each of them manually. However, benchmark() automatically performs resampling evaluations for multiple learners and tasks. benchmark_grid() creates fully crossed designs: Multiple Learners for multiple Tasks are compared w.r.t. multiple Resamplings.\n\nlearners = lrns(c(\"classif.log_reg\", \"classif.ranger\"), predict_type = \"prob\")\ngrid = benchmark_grid(\n tasks = task,\n learners = learners,\n resamplings = rsmp(\"cv\", folds = 10)\n)\nbmr = benchmark(grid)\n\nCareful, large benchmarks may take a long time! This one should take less than a minute, however. In general, we want to use parallelization to speed things up on multi-core machines. For parallelization, mlr3 relies on the future package:\n\n# future::plan(\"multisession\") # uncomment for parallelization\n\nIn the benchmark we can compare different measures. Here, we look at the misclassification rate and the AUC:\n\nmeasures = msrs(c(\"classif.ce\", \"classif.auc\"))\nperformances = bmr$aggregate(measures)\nperformances[, c(\"learner_id\", \"classif.ce\", \"classif.auc\")]\n\n learner_id classif.ce classif.auc\n1: classif.log_reg 0.238 0.7908628\n2: classif.ranger 0.229 0.8054780\n\n\nWe see that the two models perform very similarly." - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#deviating-from-hyperparameters-defaults", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#deviating-from-hyperparameters-defaults", - "title": "mlr3 Basics - German Credit", - "section": "Deviating from hyperparameters defaults", - "text": "Deviating from hyperparameters defaults\nThe previously shown techniques build the backbone of a mlr3-featured machine learning workflow. However, in most cases one would never proceed in the way we did. While many R packages have carefully selected default settings, they will not perform optimally in any scenario. Typically, we can select the values of such hyperparameters. The (hyper)parameters of a Learner can be accessed and set via its ParamSet $param_set:\n\nlearner_rf$param_set\n\n\n id class lower upper nlevels default parents value\n 1: alpha ParamDbl -Inf Inf Inf 0.5 \n 2: always.split.variables ParamUty NA NA Inf \n 3: class.weights ParamUty NA NA Inf \n 4: holdout ParamLgl NA NA 2 FALSE \n 5: importance ParamFct NA NA 4 permutation\n 6: keep.inbag ParamLgl NA NA 2 FALSE \n 7: max.depth ParamInt 0 Inf Inf \n 8: min.node.size ParamInt 1 Inf Inf \n 9: min.prop ParamDbl -Inf Inf Inf 0.1 \n10: minprop ParamDbl -Inf Inf Inf 0.1 \n11: mtry ParamInt 1 Inf Inf \n12: mtry.ratio ParamDbl 0 1 Inf \n13: num.random.splits ParamInt 1 Inf Inf 1 splitrule \n14: num.threads ParamInt 1 Inf Inf 1 1\n15: num.trees ParamInt 1 Inf Inf 500 \n16: oob.error ParamLgl NA NA 2 TRUE \n17: regularization.factor ParamUty NA NA Inf 1 \n18: regularization.usedepth ParamLgl NA NA 2 FALSE \n19: replace ParamLgl NA NA 2 TRUE \n20: respect.unordered.factors ParamFct NA NA 3 ignore \n21: sample.fraction ParamDbl 0 1 Inf \n22: save.memory ParamLgl NA NA 2 FALSE \n23: scale.permutation.importance ParamLgl NA NA 2 FALSE importance \n24: se.method ParamFct NA NA 2 infjack \n25: seed ParamInt -Inf Inf Inf \n26: split.select.weights ParamUty NA NA Inf \n27: splitrule ParamFct NA NA 3 gini \n28: verbose ParamLgl NA NA 2 TRUE \n29: write.forest ParamLgl NA NA 2 TRUE \n id class lower upper nlevels default parents value\n\nlearner_rf$param_set$values = list(verbose = FALSE)\n\nWe can choose parameters for our learners in two distinct manners. If we have prior knowledge on how the learner should be (hyper-)parameterized, the way to go would be manually entering the parameters in the parameter set. In most cases, however, we would want to tune the learner so that it can search “good” model configurations itself. For now, we only want to compare a few models.\nTo get an idea on which parameters can be manipulated, we can investigate the parameters of the original package version or look into the parameter set of the learner:\n\n## ?ranger::ranger\nas.data.table(learner_rf$param_set)[, .(id, class, lower, upper)]\n\n id class lower upper\n 1: alpha ParamDbl -Inf Inf\n 2: always.split.variables ParamUty NA NA\n 3: class.weights ParamUty NA NA\n 4: holdout ParamLgl NA NA\n 5: importance ParamFct NA NA\n 6: keep.inbag ParamLgl NA NA\n 7: max.depth ParamInt 0 Inf\n 8: min.node.size ParamInt 1 Inf\n 9: min.prop ParamDbl -Inf Inf\n10: minprop ParamDbl -Inf Inf\n11: mtry ParamInt 1 Inf\n12: mtry.ratio ParamDbl 0 1\n13: num.random.splits ParamInt 1 Inf\n14: num.threads ParamInt 1 Inf\n15: num.trees ParamInt 1 Inf\n16: oob.error ParamLgl NA NA\n17: regularization.factor ParamUty NA NA\n18: regularization.usedepth ParamLgl NA NA\n19: replace ParamLgl NA NA\n20: respect.unordered.factors ParamFct NA NA\n21: sample.fraction ParamDbl 0 1\n22: save.memory ParamLgl NA NA\n23: scale.permutation.importance ParamLgl NA NA\n24: se.method ParamFct NA NA\n25: seed ParamInt -Inf Inf\n26: split.select.weights ParamUty NA NA\n27: splitrule ParamFct NA NA\n28: verbose ParamLgl NA NA\n29: write.forest ParamLgl NA NA\n id class lower upper\n\n\nFor the random forest two meaningful parameters which steer model complexity are num.trees and mtry. num.trees defaults to 500 and mtry to floor(sqrt(ncol(data) - 1)), in our case 4.\nIn the following we aim to train three different learners:\n\nThe default random forest.\nA random forest with low num.trees and low mtry.\nA random forest with high num.trees and high mtry.\n\nWe will benchmark their performance on the German credit dataset. For this we construct the three learners and set the parameters accordingly:\n\nrf_med = lrn(\"classif.ranger\", id = \"med\", predict_type = \"prob\")\n\nrf_low = lrn(\"classif.ranger\", id = \"low\", predict_type = \"prob\",\n num.trees = 5, mtry = 2)\n\nrf_high = lrn(\"classif.ranger\", id = \"high\", predict_type = \"prob\",\n num.trees = 1000, mtry = 11)\n\nOnce the learners are defined, we can benchmark them:\n\nlearners = list(rf_low, rf_med, rf_high)\ngrid = benchmark_grid(\n tasks = task,\n learners = learners,\n resamplings = rsmp(\"cv\", folds = 10)\n)\n\n\nbmr = benchmark(grid)\nprint(bmr)\n\n of 30 rows with 3 resampling runs\n nr task_id learner_id resampling_id iters warnings errors\n 1 GermanCredit low cv 10 0 0\n 2 GermanCredit med cv 10 0 0\n 3 GermanCredit high cv 10 0 0\n\n\nWe compare misclassification rate and AUC again:\n\nmeasures = msrs(c(\"classif.ce\", \"classif.auc\"))\nperformances = bmr$aggregate(measures)\nperformances[, .(learner_id, classif.ce, classif.auc)]\n\n learner_id classif.ce classif.auc\n1: low 0.262 0.7449380\n2: med 0.236 0.8040478\n3: high 0.230 0.8030476\n\nautoplot(bmr)\n\n\n\n\n\n\n\n\nThe “low” settings seem to underfit a bit, the “high” setting is comparable to the default setting “med”." - }, - { - "objectID": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#outlook", - "href": "gallery/2020-03-11-basics-german-credit/2020-03-11-mlr3-basics-german-credit.html#outlook", - "title": "mlr3 Basics - German Credit", - "section": "Outlook", - "text": "Outlook\nThis tutorial was a detailed introduction to machine learning workflows within mlr3. Having followed this tutorial you should be able to run your first models yourself. Next to that we spiked into performance evaluation and benchmarking. Furthermore, we showed how to customize learners.\nThe next parts of the tutorial will go more into depth into additional mlr3 topics:\n\nPart II - Tuning introduces you to the mlr3tuning package\nPart III - Pipelines introduces you to the mlr3pipelines package" - }, - { - "objectID": "gallery/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/integer-hyperparameters-in-tuners-for-real-valued-search-spaces.html", - "href": "gallery/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/integer-hyperparameters-in-tuners-for-real-valued-search-spaces.html", - "title": "Integer Hyperparameters in Tuners for Real-valued Search Spaces", - "section": "", - "text": "Tuner for real-valued search spaces are not able to tune on integer hyperparameters. However, it is possible to round the real values proposed by a Tuner to integers before passing them to the learner in the evaluation. We show how to apply a parameter transformation to a ParamSet and use this set in the tuning process.\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nLoading required package: mlr3\n\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")" - }, - { - "objectID": "gallery/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/integer-hyperparameters-in-tuners-for-real-valued-search-spaces.html#task-and-learner", - "href": "gallery/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/integer-hyperparameters-in-tuners-for-real-valued-search-spaces.html#task-and-learner", - "title": "Integer Hyperparameters in Tuners for Real-valued Search Spaces", - "section": "Task and Learner", - "text": "Task and Learner\nIn this example, we use the k-Nearest-Neighbor classification learner. We want to tune the integer-valued hyperparameter k which defines the numbers of neighbors.\n\nlearner = lrn(\"classif.kknn\")\nprint(learner$param_set$params$k)\n\n id class lower upper levels default\n1: k ParamInt 1 Inf 7" - }, - { - "objectID": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html", - "href": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html", - "title": "House Prices in King County", - "section": "", - "text": "The use-case illustrated below touches on the following concepts:\nThe relevant sections in the mlr3book are linked to for the reader’s convenience.\nThis use case shows how to model housing price data in King County. Following features are illustrated:\nWe load the mlr3verse package which pulls in the most important packages for this example.\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented." - }, - { - "objectID": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html#use-case-regr-houses", - "href": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html#use-case-regr-houses", - "title": "House Prices in King County", - "section": "House Price Prediction in King County", - "text": "House Price Prediction in King County\nWe use the kc_housing dataset contained in the package mlr3data in order to provide a use-case for the application of mlr3 on real-world data.\n\ndata(\"kc_housing\", package = \"mlr3data\")\n\n\nExploratory Data Analysis\nIn order to get a quick impression of our data, we perform some initial Exploratory Data Analysis. This helps us to get a first impression of our data and might help us arrive at additional features that can help with the prediction of the house prices.\nWe can get a quick overview using R’s summary function:\n\nsummary(kc_housing)\n\n date price bedrooms bathrooms sqft_living sqft_lot \n Min. :2014-05-02 00:00:00.0 Min. : 75000 Min. : 0.000 Min. :0.000 Min. : 290 Min. : 520 \n 1st Qu.:2014-07-22 00:00:00.0 1st Qu.: 321950 1st Qu.: 3.000 1st Qu.:1.750 1st Qu.: 1427 1st Qu.: 5040 \n Median :2014-10-16 00:00:00.0 Median : 450000 Median : 3.000 Median :2.250 Median : 1910 Median : 7618 \n Mean :2014-10-29 03:58:09.9 Mean : 540088 Mean : 3.371 Mean :2.115 Mean : 2080 Mean : 15107 \n 3rd Qu.:2015-02-17 00:00:00.0 3rd Qu.: 645000 3rd Qu.: 4.000 3rd Qu.:2.500 3rd Qu.: 2550 3rd Qu.: 10688 \n Max. :2015-05-27 00:00:00.0 Max. :7700000 Max. :33.000 Max. :8.000 Max. :13540 Max. :1651359 \n \n floors waterfront view condition grade sqft_above sqft_basement \n Min. :1.000 Mode :logical Min. :0.0000 Min. :1.000 Min. : 1.000 Min. : 290 Min. : 10.0 \n 1st Qu.:1.000 FALSE:21450 1st Qu.:0.0000 1st Qu.:3.000 1st Qu.: 7.000 1st Qu.:1190 1st Qu.: 450.0 \n Median :1.500 TRUE :163 Median :0.0000 Median :3.000 Median : 7.000 Median :1560 Median : 700.0 \n Mean :1.494 Mean :0.2343 Mean :3.409 Mean : 7.657 Mean :1788 Mean : 742.4 \n 3rd Qu.:2.000 3rd Qu.:0.0000 3rd Qu.:4.000 3rd Qu.: 8.000 3rd Qu.:2210 3rd Qu.: 980.0 \n Max. :3.500 Max. :4.0000 Max. :5.000 Max. :13.000 Max. :9410 Max. :4820.0 \n NA's :13126 \n yr_built yr_renovated zipcode lat long sqft_living15 sqft_lot15 \n Min. :1900 Min. :1934 Min. :98001 Min. :47.16 Min. :-122.5 Min. : 399 Min. : 651 \n 1st Qu.:1951 1st Qu.:1987 1st Qu.:98033 1st Qu.:47.47 1st Qu.:-122.3 1st Qu.:1490 1st Qu.: 5100 \n Median :1975 Median :2000 Median :98065 Median :47.57 Median :-122.2 Median :1840 Median : 7620 \n Mean :1971 Mean :1996 Mean :98078 Mean :47.56 Mean :-122.2 Mean :1987 Mean : 12768 \n 3rd Qu.:1997 3rd Qu.:2007 3rd Qu.:98118 3rd Qu.:47.68 3rd Qu.:-122.1 3rd Qu.:2360 3rd Qu.: 10083 \n Max. :2015 Max. :2015 Max. :98199 Max. :47.78 Max. :-121.3 Max. :6210 Max. :871200 \n NA's :20699 \n\ndim(kc_housing)\n\n[1] 21613 20\n\n\nOur dataset has 21613 observations and 20 columns. The variable we want to predict is price. In addition to the price column, we have several other columns:\n\nid: A unique identifier for every house.\ndate: A date column, indicating when the house was sold. This column is currently not encoded as a date and requires some preprocessing.\nzipcode: A column indicating the ZIP code. This is a categorical variable with many factor levels.\nlong, lat The longitude and latitude of the house\n... several other numeric columns providing information about the house, such as number of rooms, square feet etc.\n\nBefore we continue with the analysis, we preprocess some features so that they are stored in the correct format.\nFirst we convert the date column to numeric. To do so, we convert the date to the POSIXct date/time class with the anytime package. Next, use difftime() to convert to days since the first day recorded in the data set:\n\nlibrary(anytime)\ndates = anytime(kc_housing$date)\nkc_housing$date = as.numeric(difftime(dates, min(dates), units = \"days\"))\n\nAfterwards, we convert the zip code to a factor:\n\nkc_housing$zipcode = as.factor(kc_housing$zipcode)\n\nAnd add a new column renovated indicating whether a house was renovated at some point.\n\nkc_housing$renovated = as.numeric(!is.na(kc_housing$yr_renovated))\nkc_housing$has_basement = as.numeric(!is.na(kc_housing$sqft_basement))\n\nWe drop the id column which provides no information about the house prices:\n\nkc_housing$id = NULL\n\nAdditionally, we convert the price from Dollar to units of 1000 Dollar to improve readability.\n\nkc_housing$price = kc_housing$price / 1000\n\nAdditionally, for now we simply drop the columns that have missing values, as some of our learners can not deal with them. A better option to deal with missing values would be imputation, i.e. replacing missing values with valid ones. We will deal with this in a separate article.\n\nkc_housing$yr_renovated = NULL\nkc_housing$sqft_basement = NULL\n\nWe can now plot the density of the price to get a first impression on its distribution.\n\nlibrary(ggplot2)\nggplot(kc_housing, aes(x = price)) + geom_density()\n\n\n\n\n\n\n\n\nWe can see that the prices for most houses lie between 75.000 and 1.5 million dollars. There are few extreme values of up to 7.7 million dollars.\nFeature engineering often allows us to incorporate additional knowledge about the data and underlying processes. This can often greatly enhance predictive performance. A simple example: A house which has yr_renovated == 0 means that is has not been renovated yet. Additionally, we want to drop features which should not have any influence (id column).\nAfter those initial manipulations, we load all required packages and create a TaskRegr containing our data.\n\ntsk = as_task_regr(kc_housing, target = \"price\")\n\nWe can inspect associations between variables using mlr3viz’s autoplot function in order to get some good first impressions for our data. Note, that this does in no way prevent us from using other powerful plot functions of our choice on the original data.\n\nDistribution of the price:\nThe outcome we want to predict is the price variable. The autoplot function provides a good first glimpse on our data. As the resulting object is a ggplot2 object, we can use faceting and other functions from ggplot2 in order to enhance plots.\n\nautoplot(tsk) + facet_wrap(~renovated)\n\n\n\n\n\n\n\n\nWe can observe that renovated flats seem to achieve higher sales values, and this might thus be a relevant feature.\nAdditionally, we can for example look at the condition of the house. Again, we clearly can see that the price rises with increasing condition.\n\nautoplot(tsk) + facet_wrap(~condition)\n\n\n\n\n\n\n\n\n\n\nAssociation between variables\nIn addition to the association with the target variable, the association between the features can also lead to interesting insights. We investigate using variables associated with the quality and size of the house. Note that we use $clone() and $select() to clone the task and select only a subset of the features for the autoplot function, as autoplot per default uses all features. The task is cloned before we select features in order to keep the original task intact.\n\n# Variables associated with quality\nautoplot(tsk$clone()$select(tsk$feature_names[c(3, 17)]), type = \"pairs\")\n\nRegistered S3 method overwritten by 'GGally':\n method from \n +.gg ggplot2\n\n\n`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.\n`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.\n\n\n\n\n\n\n\n\n\n\nautoplot(tsk$clone()$select(tsk$feature_names[c(9:12)]), type = \"pairs\")\n\n\n\n\n\n\n\n\n\n\n\nSplitting into train and test data\nIn mlr3, we do not create train and test data sets, but instead keep only a vector of train and test indices.\n\ntrain.idx = sample(seq_len(tsk$nrow), 0.7 * tsk$nrow)\ntest.idx = setdiff(seq_len(tsk$nrow), train.idx)\n\nWe can do the same for our task:\n\ntask_train = tsk$clone()$filter(train.idx)\ntask_test = tsk$clone()$filter(test.idx)\n\n\n\nA first model: Decision Tree\nDecision trees cannot only be used as a powerful tool for predictive models but also for exploratory data analysis. In order to fit a decision tree, we first get the regr.rpart learner from the mlr_learners dictionary by using the sugar function lrn.\nFor now, we leave out the zipcode variable, as we also have the latitude and longitude of each house. Again, we use $clone(), so we do not change the original task.\n\ntsk_nozip = task_train$clone()$select(setdiff(tsk$feature_names, \"zipcode\"))\n\n# Get the learner\nlrn = lrn(\"regr.rpart\")\n\n# And train on the task\nlrn$train(tsk_nozip, row_ids = train.idx)\n\n\nplot(lrn$model)\ntext(lrn$model)\n\n\n\n\n\n\n\n\nThe learned tree relies on several variables in order to distinguish between cheaper and pricier houses. The features we split along are grade, sqft_living, but also some features related to the area (longitude and latitude). We can visualize the price across different regions in order to get more info:\n\n# Load the ggmap package in order to visualize on a map\nlibrary(ggmap)\n\n# And create a quick plot for the price\nqmplot(long, lat, maptype = \"watercolor\", color = log(price),\n data = kc_housing[train.idx[1:3000],]) +\n scale_colour_viridis_c()\n\n\n\n\n\n\n\n# And the zipcode\nqmplot(long, lat, maptype = \"watercolor\", color = zipcode,\n data = kc_housing[train.idx[1:3000],]) + guides(color = FALSE)\n\nWarning: `guides( = FALSE)` is deprecated. Please use `guides( = \"none\")` instead.\n\n\n\n\n\n\n\n\n\nWe can see that the price is clearly associated with the zipcode when comparing then two plots. As a result, we might want to indeed use the zipcode column in our future endeavors.\n\n\nA first baseline: Decision Tree\nAfter getting an initial idea for our data, we might want to construct a first baseline, in order to see what a simple model already can achieve.\nWe use resample() with 3-fold cross-validation on our training data in order to get a reliable estimate of the algorithm’s performance on future data. Before we start with defining and training learners, we create a Resampling in order to make sure that we always compare on exactly the same data.\n\ncv3 = rsmp(\"cv\", folds = 3)\n\nFor the cross-validation we only use the training data by cloning the task and selecting only observations from the training set.\n\nlrn_rpart = lrn(\"regr.rpart\")\nres = resample(task = task_train, lrn_rpart, cv3)\nres$score(msr(\"regr.rmse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: kc_housing regr.rpart cv 1\n2: kc_housing regr.rpart cv 2\n3: kc_housing regr.rpart cv 3\n prediction regr.rmse\n1: 205.7541\n2: 205.6597\n3: 213.3846\n\nsprintf(\"RMSE of the simple rpart: %s\", round(sqrt(res$aggregate()), 2))\n\n[1] \"RMSE of the simple rpart: 208.3\"\n\n\n\n\nMany Trees: Random Forest\nWe might be able to improve upon the RMSE using more powerful learners. We first load the mlr3learners package, which contains the ranger learner (a package which implements the “Random Forest” algorithm).\n\nlibrary(mlr3learners)\nlrn_ranger = lrn(\"regr.ranger\", num.trees = 15L)\nres = resample(task = task_train, lrn_ranger, cv3)\nres$score(msr(\"regr.rmse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: kc_housing regr.ranger cv 1\n2: kc_housing regr.ranger cv 2\n3: kc_housing regr.ranger cv 3\n prediction regr.rmse\n1: 141.3517\n2: 159.3682\n3: 140.6830\n\nsprintf(\"RMSE of the simple ranger: %s\", round(sqrt(res$aggregate()), 2))\n\n[1] \"RMSE of the simple ranger: 147.39\"\n\n\nOften tuning RandomForest methods does not increase predictive performances substantially. If time permits, it can nonetheless lead to improvements and should thus be performed. In this case, we resort to tune a different kind of model: Gradient Boosted Decision Trees from the package xgboost.\n\n\nA better baseline: AutoTuner\nTuning can often further improve the performance. In this case, we tune the xgboost learner in order to see whether this can improve performance. For the AutoTuner we have to specify a Termination Criterion (how long the tuning should run) a Tuner (which tuning method to use) and a ParamSet (which space we might want to search through). For now, we do not use the zipcode column, as xgboost cannot naturally deal with categorical features. The AutoTuner automatically performs nested cross-validation.\n\nlrn_xgb = lrn(\"regr.xgboost\")\n\n# Define the search space\nsearch_space = ps(\n eta = p_dbl(lower = 0.2, upper = .4),\n min_child_weight = p_dbl(lower = 1, upper = 20),\n subsample = p_dbl(lower = .7, upper = .8),\n colsample_bytree = p_dbl( lower = .9, upper = 1),\n colsample_bylevel = p_dbl(lower = .5, upper = .7),\n nrounds = p_int(lower = 1L, upper = 25))\n\nat = auto_tuner(\n method = \"random_search\",\n learner = lrn_xgb,\n resampling = rsmp(\"holdout\"),\n measure = msr(\"regr.rmse\"),\n search_space = search_space,\n term_evals = 10,\n batch_size = 40)\n\n\n# And resample the AutoTuner\nres = resample(tsk_nozip, at, cv3, store_models = TRUE)\n\n\n\n\n\n\n\n\nres$score(msr(\"regr.rmse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: kc_housing regr.xgboost.tuned cv 1\n2: kc_housing regr.xgboost.tuned cv 2\n3: kc_housing regr.xgboost.tuned cv 3\n prediction regr.rmse\n1: 147.8212\n2: 131.1153\n3: 137.3479\n\nsprintf(\"RMSE of the tuned xgboost: %s\", round(sqrt(res$aggregate()), 2))\n\n[1] \"RMSE of the tuned xgboost: 138.93\"\n\n\nWe can obtain the resulting parameters in the respective splits by accessing the ResampleResult.\n\nsapply(res$learners, function(x) x$learner$param_set$values)[-2,]\n\n [,1] [,2] [,3] \nnrounds 25 17 19 \nverbose 0 0 0 \neta 0.220869 0.2749522 0.3483028\nmin_child_weight 1.885109 3.188169 5.418313 \nsubsample 0.7542127 0.7163938 0.7693532\ncolsample_bytree 0.9032271 0.9587077 0.9693665\ncolsample_bylevel 0.5259713 0.5310385 0.651549 \n\n\nNOTE: To keep runtime low, we only tune parts of the hyperparameter space of xgboost in this example. Additionally, we only allow for \\(10\\) random search iterations, which is usually too little for real-world applications. Nonetheless, we are able to obtain an improved performance when comparing to the ranger model.\nIn order to further improve our results we have several options:\n\nFind or engineer better features\nRemove Features to avoid overfitting\nObtain additional data (often prohibitive)\nTry more models\nImprove the tuning\n\nIncrease the tuning budget\nEnlarge the tuning search space\nUse a more efficient tuning algorithm\n\nStacking and Ensembles\n\nBelow we will investigate some of those possibilities and investigate whether this improves performance.\n\n\nAdvanced: Engineering Features: Mutating ZIP-Codes\nIn order to better cluster the zip codes, we compute a new feature: med_price: It computes the median price in each zip-code. This might help our model to improve the prediction. This is equivalent to impact encoding more information:\nWe can equip a learner with impact encoding using mlr3pipelines. More information on mlr3pipelines can be obtained from other posts.\n\nlrn_impact = po(\"encodeimpact\", affect_columns = selector_name(\"zipcode\")) %>>% lrn(\"regr.ranger\")\n\nAgain, we run resample() and compute the RMSE.\n\nres = resample(task = task_train, lrn_impact, cv3)\n\n\n\n\n\n\n\n\nres$score(msr(\"regr.rmse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: kc_housing encodeimpact.regr.ranger cv 1\n2: kc_housing encodeimpact.regr.ranger cv 2\n3: kc_housing encodeimpact.regr.ranger cv 3\n prediction regr.rmse\n1: 118.4878\n2: 146.2844\n3: 125.7019\n\nsprintf(\"RMSE of ranger with med_price: %s\", round(sqrt(res$aggregate()), 2))\n\n[1] \"RMSE of ranger with med_price: 130.69\"\n\n\n\n\nAdvanced: Obtaining a sparser model\nIn many cases, we might want to have a sparse model. For this purpose we can use a mlr3filters::Filter implemented in mlr3filters. This can prevent our learner from overfitting make it easier for humans to interpret models as fewer variables influence the resulting prediction.\nIn this example, we use PipeOpFilter (via po(\"filter\", ...)) to add a feature-filter before training the model. For a more in-depth insight, refer to the sections on mlr3pipelines and mlr3filters in the mlr3 book: Feature Selection and Pipelines.\n\nfilter = flt(\"mrmr\")\n\nThe resulting RMSE is slightly higher, and at the same time we only use \\(12\\) features.\n\ngraph = po(\"filter\", filter, param_vals = list(filter.nfeat = 12)) %>>% po(\"learner\", lrn(\"regr.ranger\"))\nlrn_filter = as_learner(graph)\nres = resample(task = task_train, lrn_filter, cv3)\n\n\n\n\n\n\n\n\nres$score(msr(\"regr.rmse\"))\n\n task task_id learner learner_id resampling resampling_id iteration\n1: kc_housing mrmr.regr.ranger cv 1\n2: kc_housing mrmr.regr.ranger cv 2\n3: kc_housing mrmr.regr.ranger cv 3\n prediction regr.rmse\n1: 151.8919\n2: 157.1321\n3: 149.3735\n\nsprintf(\"RMSE of ranger with filtering: %s\", round(sqrt(res$aggregate()), 2))\n\n[1] \"RMSE of ranger with filtering: 152.83\"" - }, - { - "objectID": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html#summary", - "href": "gallery/2020-01-30-house-prices-in-king-county/2020-01-30-house-prices-in-king-county.html#summary", - "title": "House Prices in King County", - "section": "Summary:", - "text": "Summary:\nWe have seen different ways to improve models with respect to our criteria by:\n\nChoosing a suitable algorithm\nChoosing good hyperparameters (tuning)\nFiltering features\nEngineering new features\n\nA combination of all the above would most likely yield an even better model. This is left as an exercise to the reader.\nThe best model we found in this example is the ranger model with the added med_price feature. In a final step, we now want to assess the model’s quality on the held-out data we stored in our task_test. In order to do so, and to prevent data leakage, we can only add the median price from the training data.\n\nlibrary(data.table)\n\ndata = task_train$data(cols = c(\"price\", \"zipcode\"))\ndata[, med_price := median(price), by = \"zipcode\"]\ntest_data = task_test$data(cols = \"zipcode\")\ntest = merge(test_data, unique(data[, .(zipcode, med_price)]), all.x = TRUE)\ntask_test$cbind(test)\n\nNow we can use the augmented task_test to predict on new data.\n\nlrn_ranger$train(task_train)\npred = lrn_ranger$predict(task_test)\npred$score(msr(\"regr.rmse\"))\n\nregr.rmse \n 141.5141" - }, - { - "objectID": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html", - "href": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html", - "title": "Select Uncorrelated Features", - "section": "", - "text": "The following example describes a situation where we aim to remove correlated features. This in essence means, that we drop features until no features have a correlation higher than a given cutoff. This is often useful when we for example want to use linear models." - }, - { - "objectID": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html#prerequisites", - "href": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html#prerequisites", - "title": "Select Uncorrelated Features", - "section": "Prerequisites", - "text": "Prerequisites\nThis tutorial assumes familiarity with the basics of mlr3pipelines. Consult the mlr3book if some aspects are not fully understandable. Additionally, we compare different cutoff values via tuning using the mlr3tuning package. Again, the mlr3book has an intro to mlr3tuning and paradox.\nThe example describes a very involved use-case, where the behavior of PipeOpSelect is manipulated via a trafo on it’s ParamSet" - }, - { - "objectID": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html#getting-started", - "href": "gallery/2020-02-25-remove-correlated-features/2020-02-25-remove-correlated-features.html#getting-started", - "title": "Select Uncorrelated Features", - "section": "Getting started", - "text": "Getting started\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nThe basic pipeline looks as follows: We use PipeOpSelect to select a set of variables followed by a rpart learner.\n\ngraph_learner = po(\"select\") %>>% lrn(\"classif.rpart\")\n\nNow we get to the magic:\nWe want to use the function caret::findCorrelation() from the caret package in order to select uncorrelated variables. This function has a cutoff parameter, that specifies the maximum correlation allowed between variables. In order to expose this variable as a numeric parameter we can tune over we specify the following ParamSet:\n\nsearch_space = ps(cutoff = p_dbl(0, 1))\n\nWe define a function select_cutoff that takes as input a Task and returns a list of features we aim to keep.\nNow we use a trafo to transform the cutoff into a set of variables, which is what PipeOpSelect can work with. Note that we use x$cutoff = NULL in order to remove the temporary parameter we introduced, as PipeOpSelect does not know what to do with it.\n\nsearch_space$trafo = function(x, param_set) {\n cutoff = x$cutoff\n x$select.selector = function(task) {\n fn = task$feature_names\n data = task$data(cols = fn)\n drop = caret::findCorrelation(cor(data), cutoff = cutoff, exact = TRUE, names = TRUE)\n setdiff(fn, drop)\n }\n x$cutoff = NULL\n x\n}\n\nIf you are not sure, you understand the trafo concept, consult the mlr3book. It has a section on the trafo concept.\nNow we tune over different values for cutoff.\n\ninstance = tune(\n method = \"grid_search\",\n task = tsk(\"iris\"),\n learner = graph_learner,\n resampling = rsmp(\"cv\", folds = 3L),\n measure = msr(\"classif.ce\"),\n search_space = search_space,\n # don't need the following line for optimization, this is for\n # demonstration that different features were selected\n store_models = TRUE)\n\nIn order to demonstrate that different cutoff values result in different features being selected, we can run the following to inspect the trained models. Note this inspects only the trained models of the first CV fold of each evaluated model. The features being excluded depends on the training data seen by the pipeline and may be different in different folds, even at the same cutoff value.\n\n\n\n\nas.data.table(instance$archive)[\n order(cutoff),\n list(cutoff, classif.ce,\n featurenames = lapply(resample_result, function(x) {\n x$learners[[1]]$model$classif.rpart$train_task$feature_names\n }\n ))]\n\n cutoff classif.ce featurenames\n 1: 0.0000000 0.28666667 Sepal.Length\n 2: 0.1111111 0.28666667 Sepal.Length\n 3: 0.2222222 0.28666667 Sepal.Length\n 4: 0.3333333 0.27333333 Sepal.Length,Sepal.Width\n 5: 0.4444444 0.27333333 Sepal.Length,Sepal.Width\n 6: 0.5555556 0.27333333 Sepal.Length,Sepal.Width\n 7: 0.6666667 0.27333333 Sepal.Length,Sepal.Width\n 8: 0.7777778 0.27333333 Sepal.Length,Sepal.Width\n 9: 0.8888889 0.04000000 Petal.Width,Sepal.Length,Sepal.Width\n10: 1.0000000 0.06666667 Petal.Length,Petal.Width,Sepal.Length,Sepal.Width\n\n\nVoila, we created our own PipeOp, that uses very advanced knowledge of mlr3pipelines and paradox in only few lines of code." - }, - { - "objectID": "gallery/2021-03-10-practical-tuning-series-tune-a-preprocessing-pipeline/practical-tuning-series-tune-a-preprocessing-pipeline.html", - "href": "gallery/2021-03-10-practical-tuning-series-tune-a-preprocessing-pipeline/practical-tuning-series-tune-a-preprocessing-pipeline.html", - "title": "Practical Tuning Series - Tune a Preprocessing Pipeline", - "section": "", - "text": "Scope\nThis is the second part of the practical tuning series. The other parts can be found here:\n\nPart I - Tune a Support Vector Machine\nPart III - Build an Automated Machine Learning System\nPart IV - Tuning and Parallel Processing\n\nIn this post, we build a simple preprocessing pipeline and tune it. For this, we are using the mlr3pipelines extension package. First, we start by imputing missing values in the Pima Indians Diabetes data set. After that, we encode a factor column to numerical dummy columns in the data set. Next, we combine both preprocessing steps to a Graph and create a GraphLearner. Finally, nested resampling is used to compare the performance of two imputation methods.\n\n\nPrerequisites\nWe load the mlr3verse package which pulls in the most important packages for this example.\n\nlibrary(mlr3verse)\n\nWe initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented. The lgr package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning ).\n\nset.seed(7832)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nlgr::get_logger(\"bbotk\")$set_threshold(\"warn\")\n\nIn this example, we use the Pima Indians Diabetes data set which is used to predict whether or not a patient has diabetes. The patients are characterized by 8 numeric features of which some have missing values. We alter the data set by categorizing the feature pressure (blood pressure) into the categories \"low\", \"mid\", and \"high\".\n\n# retrieve the task from mlr3\ntask = tsk(\"pima\")\n\n# create data frame with categorized pressure feature\ndata = task$data(cols = \"pressure\")\nbreaks = quantile(data$pressure, probs = c(0, 0.33, 0.66, 1), na.rm = TRUE)\ndata$pressure = cut(data$pressure, breaks, labels = c(\"low\", \"mid\", \"high\"))\n\n# overwrite the feature in the task\ntask$cbind(data)\n\n# generate a quick textual overview\nskimr::skim(task$data())\n\n\nData summary\n\n\nName\ntask$data()\n\n\nNumber of rows\n768\n\n\nNumber of columns\n9\n\n\nKey\nNULL\n\n\n_______________________\n\n\n\nColumn type frequency:\n\n\n\nfactor\n2\n\n\nnumeric\n7\n\n\n________________________\n\n\n\nGroup variables\nNone\n\n\n\nVariable type: factor\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nordered\nn_unique\ntop_counts\n\n\n\n\ndiabetes\n0\n1.00\nFALSE\n2\nneg: 500, pos: 268\n\n\npressure\n36\n0.95\nFALSE\n3\nlow: 282, mid: 245, hig: 205\n\n\n\nVariable type: numeric\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nskim_variable\nn_missing\ncomplete_rate\nmean\nsd\np0\np25\np50\np75\np100\nhist\n\n\n\n\nage\n0\n1.00\n33.24\n11.76\n21.00\n24.00\n29.00\n41.00\n81.00\n▇▃▁▁▁\n\n\nglucose\n5\n0.99\n121.69\n30.54\n44.00\n99.00\n117.00\n141.00\n199.00\n▁▇▇▃▂\n\n\ninsulin\n374\n0.51\n155.55\n118.78\n14.00\n76.25\n125.00\n190.00\n846.00\n▇▂▁▁▁\n\n\nmass\n11\n0.99\n32.46\n6.92\n18.20\n27.50\n32.30\n36.60\n67.10\n▅▇▃▁▁\n\n\npedigree\n0\n1.00\n0.47\n0.33\n0.08\n0.24\n0.37\n0.63\n2.42\n▇▃▁▁▁\n\n\npregnant\n0\n1.00\n3.85\n3.37\n0.00\n1.00\n3.00\n6.00\n17.00\n▇▃▂▁▁\n\n\ntriceps\n227\n0.70\n29.15\n10.48\n7.00\n22.00\n29.00\n36.00\n99.00\n▆▇▁▁▁\n\n\n\n\n\nWe choose the xgboost algorithm from the xgboost package as learner.\n\nlearner = lrn(\"classif.xgboost\", nrounds = 100, id = \"xgboost\", verbose = 0)\n\n\n\nMissing Values\nThe task has missing data in five columns.\n\nround(task$missings() / task$nrow, 2)\n\ndiabetes age glucose insulin mass pedigree pregnant pressure triceps \n 0.00 0.00 0.01 0.49 0.01 0.00 0.00 0.05 0.30 \n\n\nThe xgboost learner has an internal method for handling missing data but some learners cannot handle missing values. We will try to beat the internal method in terms of predictive performance. The mlr3pipelines package offers various methods to impute missing values.\n\nmlr_pipeops$keys(\"^impute\")\n\n[1] \"imputeconstant\" \"imputehist\" \"imputelearner\" \"imputemean\" \"imputemedian\" \"imputemode\" \n[7] \"imputeoor\" \"imputesample\" \n\n\nWe choose the PipeOpImputeOOR that adds the new factor level \".MISSING\". to factorial features and imputes numerical features by constant values shifted below the minimum (default) or above the maximum.\n\nimputer = po(\"imputeoor\")\nprint(imputer)\n\nPipeOp: (not trained)\nvalues: \nInput channels :\n input [Task,Task]\nOutput channels :\n output [Task,Task]\n\n\nAs the output suggests, the in- and output of this pipe operator is a Task for both the training and the predict step. We can manually train the pipe operator to check its functionality:\n\ntask_imputed = imputer$train(list(task))[[1]]\ntask_imputed$missings()\n\ndiabetes age pedigree pregnant glucose insulin mass pressure triceps \n 0 0 0 0 0 0 0 0 0 \n\n\nLet’s compare an observation with missing values to the observation with imputed observation.\n\nrbind(\n task$data()[8,],\n task_imputed$data()[8,]\n)\n\n diabetes age glucose insulin mass pedigree pregnant pressure triceps\n1: neg 29 115 NA 35.3 0.134 10 NA\n2: neg 29 115 -819 35.3 0.134 10 .MISSING -86\n\n\nNote that OOR imputation is in particular useful for tree-based models, but should not be used for linear models or distance-based models.\n\n\nFactor Encoding\nThe xgboost learner cannot handle categorical features. Therefore, we must to convert factor columns to numerical dummy columns. For this, we argument the xgboost learner with automatic factor encoding.\nThe PipeOpEncode encodes factor columns with one of six methods. In this example, we use one-hot encoding which creates a new binary column for each factor level.\n\nfactor_encoding = po(\"encode\", method = \"one-hot\")\n\nWe manually trigger the encoding on the task.\n\nfactor_encoding$train(list(task))\n\n$output\n (768 x 11): Pima Indian Diabetes\n* Target: diabetes\n* Properties: twoclass\n* Features (10):\n - dbl (10): age, glucose, insulin, mass, pedigree, pregnant, pressure.high, pressure.low, pressure.mid,\n triceps\n\n\nThe factor column pressure has been converted to the three binary columns \"pressure.low\", \"pressure.mid\", and \"pressure.high\".\n\n\nConstructing the Pipeline\nWe created two preprocessing steps which could be used to create a new task with encoded factor variables and imputed missing values. However, if we do this before resampling, information from the test can leak into our training step which typically leads to overoptimistic performance measures. To avoid this, we add the preprocessing steps to the Learner itself, creating a GraphLearner. For this, we create a Graph first.\n\ngraph = po(\"encode\") %>>%\n po(\"imputeoor\") %>>%\n learner\nplot(graph, html = TRUE)\n\n\n\n\n\nWe use as_learner() to wrap the Graph into a GraphLearner with which allows us to use the graph like a normal learner.\n\ngraph_learner = as_learner(graph)\n\n# short learner id for printing\ngraph_learner$id = \"graph_learner\"\n\nThe GraphLearner can be trained and used for making predictions. Instead of calling $train() or $predict() manually, we will directly use it for resampling. We choose a 3-fold cross-validation as the resampling strategy.\n\nresampling = rsmp(\"cv\", folds = 3)\n\nrr = resample(task = task, learner = graph_learner, resampling = resampling)\n\n\nrr$score()[, c(\"iteration\", \"task_id\", \"learner_id\", \"resampling_id\", \"classif.ce\"), with = FALSE]\n\n iteration task_id learner_id resampling_id classif.ce\n1: 1 pima graph_learner cv 0.2851562\n2: 2 pima graph_learner cv 0.2460938\n3: 3 pima graph_learner cv 0.2968750\n\n\nFor each resampling iteration, the following steps are performed:\n\nThe task is subsetted to the training indices.\nThe factor encoder replaces factor features with dummy columns in the training task.\nThe OOR imputer determines values to impute from the training task and then replaces all missing values with learned imputation values.\nThe learner is applied on the modified training task and the model is stored inside the learner.\n\nNext is the predict step:\n\nThe task is subsetted to the test indices.\nThe factor encoder replaces all factor features with dummy columns in the test task.\nThe OOR imputer replaces all missing values of the test task with the imputation values learned on the training set.\nThe learner’s predict method is applied on the modified test task.\n\nBy following this procedure, it is guaranteed that no information can leak from the training step to the predict step.\n\n\nTuning the Pipeline\nLet’s have a look at the parameter set of the GraphLearner. It consists of the xgboost hyperparameters, and additionally, the parameter of the PipeOp encode and imputeoor. All hyperparameters are prefixed with the id of the respective PipeOp or learner.\n\nas.data.table(graph_learner$param_set)[, c(\"id\", \"class\", \"lower\", \"upper\", \"nlevels\"), with = FALSE]\n\n id class lower upper nlevels\n 1: encode.method ParamFct NA NA 5\n 2: encode.affect_columns ParamUty NA NA Inf\n 3: imputeoor.min ParamLgl NA NA 2\n 4: imputeoor.offset ParamDbl 0 Inf Inf\n 5: imputeoor.multiplier ParamDbl 0 Inf Inf\n 6: imputeoor.affect_columns ParamUty NA NA Inf\n 7: xgboost.alpha ParamDbl 0 Inf Inf\n 8: xgboost.approxcontrib ParamLgl NA NA 2\n 9: xgboost.base_score ParamDbl -Inf Inf Inf\n10: xgboost.booster ParamFct NA NA 3\n11: xgboost.callbacks ParamUty NA NA Inf\n12: xgboost.colsample_bylevel ParamDbl 0 1 Inf\n13: xgboost.colsample_bynode ParamDbl 0 1 Inf\n14: xgboost.colsample_bytree ParamDbl 0 1 Inf\n15: xgboost.disable_default_eval_metric ParamLgl NA NA 2\n16: xgboost.early_stopping_rounds ParamInt 1 Inf Inf\n17: xgboost.eta ParamDbl 0 1 Inf\n18: xgboost.eval_metric ParamUty NA NA Inf\n19: xgboost.feature_selector ParamFct NA NA 5\n20: xgboost.feval ParamUty NA NA Inf\n21: xgboost.gamma ParamDbl 0 Inf Inf\n22: xgboost.grow_policy ParamFct NA NA 2\n23: xgboost.interaction_constraints ParamUty NA NA Inf\n24: xgboost.iterationrange ParamUty NA NA Inf\n25: xgboost.lambda ParamDbl 0 Inf Inf\n26: xgboost.lambda_bias ParamDbl 0 Inf Inf\n27: xgboost.max_bin ParamInt 2 Inf Inf\n28: xgboost.max_delta_step ParamDbl 0 Inf Inf\n29: xgboost.max_depth ParamInt 0 Inf Inf\n30: xgboost.max_leaves ParamInt 0 Inf Inf\n31: xgboost.maximize ParamLgl NA NA 2\n32: xgboost.min_child_weight ParamDbl 0 Inf Inf\n33: xgboost.missing ParamDbl -Inf Inf Inf\n34: xgboost.monotone_constraints ParamUty NA NA Inf\n35: xgboost.normalize_type ParamFct NA NA 2\n36: xgboost.nrounds ParamInt 1 Inf Inf\n37: xgboost.nthread ParamInt 1 Inf Inf\n38: xgboost.ntreelimit ParamInt 1 Inf Inf\n39: xgboost.num_parallel_tree ParamInt 1 Inf Inf\n40: xgboost.objective ParamUty NA NA Inf\n41: xgboost.one_drop ParamLgl NA NA 2\n42: xgboost.outputmargin ParamLgl NA NA 2\n43: xgboost.predcontrib ParamLgl NA NA 2\n44: xgboost.predictor ParamFct NA NA 2\n45: xgboost.predinteraction ParamLgl NA NA 2\n46: xgboost.predleaf ParamLgl NA NA 2\n47: xgboost.print_every_n ParamInt 1 Inf Inf\n48: xgboost.process_type ParamFct NA NA 2\n49: xgboost.rate_drop ParamDbl 0 1 Inf\n50: xgboost.refresh_leaf ParamLgl NA NA 2\n51: xgboost.reshape ParamLgl NA NA 2\n52: xgboost.seed_per_iteration ParamLgl NA NA 2\n53: xgboost.sampling_method ParamFct NA NA 2\n54: xgboost.sample_type ParamFct NA NA 2\n55: xgboost.save_name ParamUty NA NA Inf\n56: xgboost.save_period ParamInt 0 Inf Inf\n57: xgboost.scale_pos_weight ParamDbl -Inf Inf Inf\n58: xgboost.skip_drop ParamDbl 0 1 Inf\n59: xgboost.strict_shape ParamLgl NA NA 2\n60: xgboost.subsample ParamDbl 0 1 Inf\n61: xgboost.top_k ParamInt 0 Inf Inf\n62: xgboost.training ParamLgl NA NA 2\n63: xgboost.tree_method ParamFct NA NA 5\n64: xgboost.tweedie_variance_power ParamDbl 1 2 Inf\n65: xgboost.updater ParamUty NA NA Inf\n66: xgboost.verbose ParamInt 0 2 3\n67: xgboost.watchlist ParamUty NA NA Inf\n68: xgboost.xgb_model ParamUty NA NA Inf\n id class lower upper nlevels\n\n\nWe will tune the encode method.\n\ngraph_learner$param_set$values$encode.method = to_tune(c(\"one-hot\", \"treatment\"))\n\nWe define a tuning instance and use grid search since we want to try all encode methods.\n\ninstance = tune(\n method = \"grid_search\",\n task = task,\n learner = graph_learner,\n resampling = rsmp(\"cv\", folds = 3),\n measure = msr(\"classif.ce\")\n)\n\nThe archive shows us the performance of the model with different encoding methods.\n\nprint(instance$archive)\n\n\n encode.method classif.ce runtime_learners timestamp batch_nr warnings errors resample_result\n1: one-hot 0.27 0.52 2022-09-17 18:42:23.10 1 0 0 \n2: treatment 0.27 0.41 2022-09-17 18:42:23.56 2 0 0 \n\n\n\n\nNested Resampling\nWe create one GraphLearner with imputeoor and test it against a GraphLearner that uses the internal imputation method of xgboost. Applying nested resampling ensures a fair comparison of the predictive performances.\n\ngraph_1 = po(\"encode\") %>>%\n learner\ngraph_learner_1 = GraphLearner$new(graph_1)\n\ngraph_learner_1$param_set$values$encode.method = to_tune(c(\"one-hot\", \"treatment\"))\n\nat_1 = AutoTuner$new(\n learner = graph_learner_1,\n resampling = resampling,\n measure = msr(\"classif.ce\"),\n terminator = trm(\"none\"),\n tuner = tnr(\"grid_search\"),\n store_models = TRUE\n)\n\n\ngraph_2 = po(\"encode\") %>>%\n po(\"imputeoor\") %>>%\n learner\ngraph_learner_2 = GraphLearner$new(graph_2)\n\ngraph_learner_2$param_set$values$encode.method = to_tune(c(\"one-hot\", \"treatment\"))\n\nat_2 = AutoTuner$new(\n learner = graph_learner_2,\n resampling = resampling,\n measure = msr(\"classif.ce\"),\n terminator = trm(\"none\"),\n tuner = tnr(\"grid_search\"),\n store_models = TRUE\n)\n\nWe run the benchmark.\n\nresampling_outer = rsmp(\"cv\", folds = 3)\ndesign = benchmark_grid(task, list(at_1, at_2), resampling_outer)\n\nbmr = benchmark(design, store_models = TRUE)\n\n\n\n\n\n\n\nWe compare the aggregated performances on the outer test sets which give us an unbiased performance estimate of the GraphLearners with the different encoding methods.\n\nbmr$aggregate()\n\n nr resample_result task_id learner_id resampling_id iters classif.ce\n1: 1 pima encode.xgboost.tuned cv 3 0.2695312\n2: 2 pima encode.imputeoor.xgboost.tuned cv 3 0.2682292\n\nautoplot(bmr)\n\n\n\n\n\n\n\n\nNote that in practice, it is required to tune preprocessing hyperparameters jointly with the hyperparameters of the learner. Otherwise, comparing preprocessing steps is not feasible and can lead to wrong conclusions.\nApplying nested resampling can be shortened by using the auto_tuner()-shortcut.\n\ngraph_1 = po(\"encode\") %>>% learner\ngraph_learner_1 = as_learner(graph_1)\ngraph_learner_1$param_set$values$encode.method = to_tune(c(\"one-hot\", \"treatment\"))\n\nat_1 = auto_tuner(\n method = \"grid_search\",\n learner = graph_learner_1,\n resampling = resampling,\n measure = msr(\"classif.ce\"),\n store_models = TRUE)\n\ngraph_2 = po(\"encode\") %>>% po(\"imputeoor\") %>>% learner\ngraph_learner_2 = as_learner(graph_2)\ngraph_learner_2$param_set$values$encode.method = to_tune(c(\"one-hot\", \"treatment\"))\n\nat_2 = auto_tuner(\n method = \"grid_search\",\n learner = graph_learner_2,\n resampling = resampling,\n measure = msr(\"classif.ce\"),\n store_models = TRUE)\n\ndesign = benchmark_grid(task, list(at_1, at_2), rsmp(\"cv\", folds = 3))\n\nbmr = benchmark(design, store_models = TRUE)\n\n\n\nFinal Model\nWe train the chosen GraphLearner with the AutoTuner to get a final model with optimized hyperparameters.\n\nat_2$train(task)\n\nThe trained model can now be used to make predictions on new data at_2$predict(). The pipeline ensures that the preprocessing is always a part of the train and predict step.\n\n\nResources\nThe mlr3book includes chapters on pipelines and hyperparameter tuning. The mlr3cheatsheets contain frequently used commands and workflows of mlr3." - }, - { - "objectID": "team.html", - "href": "team.html", - "title": "", - "section": "", - "text": "Bernd Bischl\n\nProfessor of Statistical Learning and Data Science at the LMU Munich. I created mlr a long time ago at the beginning of my PhD. Nowadays, I spend most of my time in project supervision, code reviews and helping to design new parts of the framework. I was part of the design process of nearly all parts of the new mlr3, but nearly all code has been written by the other developers.\n\n\n\n\n\n\n\n\nMichel Lang\n\nPostdoc at the TU Dortmund and one of the main developers of mlr. I've worked on many internal parts of mlr and started to implement support for survival analysis. Now main developer of mlr3.\n\n\n\n\n\n\n\n\nMarc Becker\n\nResearch engineer at the LMU Munich and main developer of the mlr3 optimization packages.\n\n\n\n\n\n\n\n\nRaphael Sonabend\n\nPostdoc at Imperial College London. I am the main developer of mlr3proba and also the previous maintainer of mlr3extralearners.\n\n\n\n\n\n\n\n\nPatrick Schratz\n\nR consultant in Zurich, Switzerland. PhD Candidate in environmental modeling. Mainly contributing to spatiotemporal packages. Maintainer of the old mlr package.\n\n\n\n\n\n\n\n\nSebastian Fischer\n\nResearch Engineer at LMU Munich. Is working on mlr3torch, mlr3oml and maintains mlr3extralearners.\n\n\n\n\n\n\n\n\nLars Kotthoff\n\nComputer Science Professor at University of Wyoming, contributes small pieces here and there.\n\n\n\n\n\n\n\n\nFlorian Pfisterer\n\nPhD Student at LMU Munich. I am interested in projects on the intersection of Meta-Learning, AutoML and Algorithmic Fairness. Mainly working on mlr3pipelines and mlr3keras/mlr3torch" - }, - { - "objectID": "index.html", - "href": "index.html", - "title": "", - "section": "", - "text": "An open-source collection of R packages providing a unified interface for machine learning in the R language." - }, - { - "objectID": "index.html#resources", - "href": "index.html#resources", - "title": "", - "section": "Resources", - "text": "Resources\n\n\n\n\n\nGitHub\n\n\n\nMattermost\n\n\n\nBook\n\n\n\nStackOverflow" - }, - { - "objectID": "index.html#quick-start", - "href": "index.html#quick-start", - "title": "", - "section": "Quick Start", - "text": "Quick Start\nThe mlr3verse meta-package installs mlr3 and some of the most important extension packages:\n\ninstall.packages(\"mlr3verse\")" - }, - { - "objectID": "index.html#examples", - "href": "index.html#examples", - "title": "", - "section": "Examples", - "text": "Examples\n\n\n\n\n \n \n \n Basic machine learning\n \n \n \n Learner\n Task\n Resampling\n\n\n \n \n \n \n \n Hyperparameter tuning\n \n \n \n Tuning\n Tuning spaces\n Nested resampling\n\n\n \n \n\n\nNo matching items" - }, - { - "objectID": "pipeops.html", - "href": "pipeops.html", - "title": "", - "section": "", - "text": "Pipeline Operators\nObjects of class PipeOp are the building blocks to compose linear machine learning pipelines and non-linear Graphs. The base objects are implemented in mlr3pipelines." - }, - { - "objectID": "gallery.html", - "href": "gallery.html", - "title": "Gallery", - "section": "", - "text": "Order By\n Default\n \n Date - Oldest\n \n \n Date - Newest\n \n \n Title\n \n \n Author\n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\nTitle\n\n\nAuthor\n\n\nDate\n\n\n\n\n\n\n\n\n\nIntroduction to mlr3tuningspaces\n\n\nMarc Becker\n\n\n2021-07-06\n\n\n\n\n\n\n\nPractical Tuning Series - Tuning and Parallel Processing\n\n\nMarc Becker, Theresa Ullmann, Michel Lang, Bernd Bischl, Jakob Richter, Martin Binder\n\n\n2021-03-12\n\n\n\n\n\n\n\nPractical Tuning Series - Build an Automated Machine Learning System\n\n\nMarc Becker, Theresa Ullmann, Michel Lang, Bernd Bischl, Jakob Richter, Martin Binder\n\n\n2021-03-11\n\n\n\n\n\n\n\nPractical Tuning Series - Tune a Preprocessing Pipeline\n\n\nMarc Becker, Theresa Ullmann, Michel Lang, Bernd Bischl, Jakob Richter, Martin Binder\n\n\n2021-03-10\n\n\n\n\n\n\n\nPractical Tuning Series - Tune a Support Vector Machine\n\n\nMarc Becker, Theresa Ullmann, Michel Lang, Bernd Bischl, Jakob Richter, Martin Binder\n\n\n2021-03-09\n\n\n\n\n\n\n\nTuning a Complex Graph\n\n\nLennart Schneider\n\n\n2021-02-03\n\n\n\n\n\n\n\nInteger Hyperparameters in Tuners for Real-valued Search Spaces\n\n\nMarc Becker\n\n\n2021-01-19\n\n\n\n\n\n\n\nFeature Selection on the Titanic Data Set\n\n\nMarc Becker\n\n\n2021-01-08\n\n\n\n\n\n\n\nThreshold Tuning for Classification Tasks\n\n\nFlorian Pfisterer\n\n\n2020-10-14\n\n\n\n\n\n\n\nLiver Patient Classification Based on Diagnostic Measures\n\n\nJulian Lange, Jae-Eun Nam, Viet Tran, Simon Wiegrebe, Henri Funk (Editor)\n\n\n2020-09-11\n\n\n\n\n\n\n\nIntroduction to mlr3keras - Boston Housing\n\n\nFlorian Pfisterer\n\n\n2020-09-11\n\n\n\n\n\n\n\nComparison of Decision Boundaries of Classification Learners\n\n\nMichel Lang\n\n\n2020-08-14\n\n\n\n\n\n\n\nA Production Example Using Plumber and Docker\n\n\nLennart Schneider\n\n\n2020-08-13\n\n\n\n\n\n\n\nTarget Transformations via Pipelines\n\n\nLennart Schneider\n\n\n2020-06-15\n\n\n\n\n\n\n\nWhy R? Webinar - Pipelines and AutoML with mlr3\n\n\nBernd Bischl\n\n\n2020-05-28\n\n\n\n\n\n\n\nmlr3 and OpenML - Moneyball Use Case\n\n\nPhilipp Kopper\n\n\n2020-05-04\n\n\n\n\n\n\n\nFeature Engineering of Date-Time Variables\n\n\nLennart Schneider\n\n\n2020-05-02\n\n\n\n\n\n\n\nTuning a Stacked Learner\n\n\nMilan Dragicevic, Giuseppe Casalicchio\n\n\n2020-04-27\n\n\n\n\n\n\n\nA Pipeline for the Titanic Data Set - Advanced\n\n\nFlorian Pfisterer\n\n\n2020-04-27\n\n\n\n\n\n\n\nPipelines, Selectors, Branches\n\n\nMilan Dragicevic, Giuseppe Casalicchio\n\n\n2020-04-23\n\n\n\n\n\n\n\nRegression Chains\n\n\nLennart Schneider\n\n\n2020-04-18\n\n\n\n\n\n\n\nResampling - Stratified, Blocked and Predefined\n\n\nMilan Dragicevic, Giuseppe Casalicchio\n\n\n2020-03-30\n\n\n\n\n\n\n\nImbalanced Data Handling with mlr3\n\n\nGiuseppe Casalicchio\n\n\n2020-03-30\n\n\n\n\n\n\n\nmlr3 Basics on “Iris” - Hello World!\n\n\nBernd Bischl\n\n\n2020-03-18\n\n\n\n\n\n\n\nA Pipeline for the Titanic Data Set - Basics\n\n\nFlorian Pfisterer\n\n\n2020-03-12\n\n\n\n\n\n\n\nmlr3tuning Tutorial - German Credit\n\n\nMartin Binder, Florian Pfisterer\n\n\n2020-03-11\n\n\n\n\n\n\n\nmlr3pipelines Tutorial - German Credit\n\n\nMartin Binder, Florian Pfisterer\n\n\n2020-03-11\n\n\n\n\n\n\n\nmlr3 Basics - German Credit\n\n\nMartin Binder, Florian Pfisterer, Michel Lang\n\n\n2020-03-11\n\n\n\n\n\n\n\nSelect Uncorrelated Features\n\n\nMartin Binder, Florian Pfisterer\n\n\n2020-02-25\n\n\n\n\n\n\n\nTuning Over Multiple Learners\n\n\nJakob Richter, Bernd Bischl\n\n\n2020-02-01\n\n\n\n\n\n\n\nEncode Factor Levels for xgboost\n\n\nMichel Lang\n\n\n2020-01-31\n\n\n\n\n\n\n\nImpute Missing Variables\n\n\nFlorian Pfisterer\n\n\n2020-01-31\n\n\n\n\n\n\n\nHouse Prices in King County\n\n\nFlorian Pfisterer\n\n\n2020-01-30\n\n\n\n\n\n\n\nXI Jornadas de Usuarios de R - mlr3\n\n\nBernd Bischl\n\n\n2019-11-15\n\n\n\n\n\n\n\nuseR! 2019 Presentation - mlr3\n\n\nMichel Lang\n\n\n2019-08-03\n\n\n\n\n\n\n\nuseR! 2019 Presentation - mlr3pipelines\n\n\nBernd Bischl\n\n\n2019-08-03\n\n\n\n\n\n\nNo matching items" - }, - { - "objectID": "resamplings.html", - "href": "resamplings.html", - "title": "", - "section": "", - "text": "Resamplings split the observations multiple times into two sets: training and test. The former is used to fit the model, the latter is used to evaluate the predictions. The Resampling objects provide an abstraction for this procedure while respecting stratification as well as grouping/blocking if this is required by the Task.\nIf only a single split is required (i.e., a holdout split), the partition() function provides a single split into training and test set.\n\n\n\n\n\n\n\n\n\nFit a Random Forest on the Wisconsin Breast Cancer Data Set using a 3-fold cross validation.\n\nlibrary(\"mlr3verse\")\n\n# retrieve the task\ntask = tsk(\"breast_cancer\")\n\n# retrieve a learner\nlearner = lrn(\"classif.ranger\")\n\n# retrieve resampling strategy\nresampling = rsmp(\"cv\", folds = 3)\n\n# perform resampling\nrr = resample(task, learner, resampling)\n\nINFO [18:42:39.773] [mlr3] Applying learner 'classif.ranger' on task 'breast_cancer' (iter 1/3) \nINFO [18:42:40.317] [mlr3] Applying learner 'classif.ranger' on task 'breast_cancer' (iter 2/3) \nINFO [18:42:40.374] [mlr3] Applying learner 'classif.ranger' on task 'breast_cancer' (iter 3/3) \n\nrr\n\n of 3 iterations\n* Task: breast_cancer\n* Learner: classif.ranger\n* Warnings: 0 in 0 iterations\n* Errors: 0 in 0 iterations" - }, - { - "objectID": "tuners.html", - "href": "tuners.html", - "title": "", - "section": "", - "text": "Popular black-box optimization techniques are implemented in the bbotk package. The corresponding connectors to for tuning hyperparameters of learners or pipelines reside as Tuner objects in package mlr3tuning. Additionally, packages mlr3hyperband and mlr3mbo provide some modern and sophisticated approaches.\nAll tuners operator on box-constrained tuning spaces which have to be defined by the user. Some popular spaces from literature are readily available as tuning spaces.\n\n\n\n\n\n\n\n\n\nTune the hyperparameters of a classification tree on the Palmer Penguins data set with random search.\n\nlibrary(mlr3verse)\n\n# retrieve task\ntask = tsk(\"penguins\")\n\n# load learner and set search space\nlearner = lrn(\"classif.rpart\",\n cp = to_tune(1e-04, 1e-1, logscale = TRUE),\n minsplit = to_tune(2, 128, logscale = TRUE)\n)\n\n# load tuner and set batch size\ntuner = tnr(\"random_search\", batch_size = 10)\n\n# hyperparameter tuning on the palmer penguins data set\ninstance = tune(\n method = tuner,\n task = task,\n learner = learner,\n resampling = rsmp(\"holdout\"),\n measure = msr(\"classif.ce\"),\n term_evals = 50\n)\n\n# best performing hyperparameter configuration\ninstance$result\n\n cp minsplit learner_param_vals x_domain classif.ce\n1: -8.451495 0.8451619 0.02608696\n\n# surface plot\nautoplot(instance, type = \"surface\")\n\n\n\n# fit final model on complete data set\nlearner$param_set$values = instance$result_learner_param_vals\nlearner$train(task)\n\nprint(learner)\n\n: Classification Tree\n* Model: rpart\n* Parameters: xval=0, cp=0.0002136, minsplit=2\n* Packages: mlr3, rpart\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, multiclass, selected_features,\n twoclass, weights" - }, - { - "objectID": "blogroll.html", - "href": "blogroll.html", - "title": "Blogroll", - "section": "", - "text": "https://www.r-bloggers.com/" - }, - { - "objectID": "team/about-sebastian.html", - "href": "team/about-sebastian.html", - "title": "Sebastian Fischer", - "section": "", - "text": "Research Engineer at LMU Munich. Working on mlr3torch, mlr3oml and maintaining mlr3extralearners." - }, - { - "objectID": "team/about-bernd.html", - "href": "team/about-bernd.html", - "title": "Bernd Bischl", - "section": "", - "text": "Professor of Statistical Learning and Data Science at the LMU Munich. I created mlr a long time ago at the beginning of my PhD. Nowadays, I spend most of my time in project supervision, code reviews and helping to design new parts of the framework. I was part of the design process of nearly all parts of the new mlr3, but nearly all code has been written by the other developers." - }, - { - "objectID": "team/about-lars.html", - "href": "team/about-lars.html", - "title": "Lars Kotthoff", - "section": "", - "text": "Computer Science Professor at University of Wyoming, contributes small pieces here and there." - }, - { - "objectID": "team/about-patrick.html", - "href": "team/about-patrick.html", - "title": "Patrick Schratz", - "section": "", - "text": "R consultant in Zurich, Switzerland. PhD Candidate in environmental modeling. Mainly contributing to spatiotemporal packages. Maintainer of the old mlr package." - }, - { - "objectID": "team/about-raphael.html", - "href": "team/about-raphael.html", - "title": "Raphael Sonabend", - "section": "", - "text": "Postdoc at Imperial College London. I am the main developer of mlr3proba and also the previous maintainer of mlr3extralearners." - }, - { - "objectID": "team/about-florian.html", - "href": "team/about-florian.html", - "title": "Florian Pfisterer", - "section": "", - "text": "PhD Student at LMU Munich. I am interested in projects on the intersection of Meta-Learning, AutoML and Algorithmic Fairness. Mainly working on mlr3pipelines and mlr3keras/mlr3torch." - }, - { - "objectID": "team/about-michel.html", - "href": "team/about-michel.html", - "title": "Michel Lang", - "section": "", - "text": "Postdoc at the TU Dortmund and one of the main developers of mlr. I’ve worked on many internal parts of mlr and started to implement support for survival analysis. Now main developer of mlr3." - }, - { - "objectID": "team/about-marc.html", - "href": "team/about-marc.html", - "title": "Marc Becker", - "section": "", - "text": "Research engineer at the LMU Munich and main developer of the mlr3 optimization packages." - }, - { - "objectID": "packages.html", - "href": "packages.html", - "title": "Packages", - "section": "", - "text": "The mlr3 ecosystem is a collection of R packages for machine learning. The base package mlr3 only provides the basic building blocks for machine learning. The extensions packages extent mlr3 with functionality for additional task types, learning algorithms, tuning algorithms, feature selection strategies, visualizations or preprocessing capabilities. The packages are listed bellow with a short description. For more information about the packages, check out their respective homepages.\nThe dot next to the package name indicates the lifecycle stage.\n\nPackages with a green dot are stable.\nExperimental packages are marked with an orange dot .\nPlanned packages are marked with a red dot .\n\n\n\n\nGraph of Extension Packages\n\n\nIf you use our packages in your research, please cite our articles on mlr3 (Lang et al. 2019), mlr3proba (Sonabend et al. 2021) or mlr3pipelines (Binder et al. 2021). To get the citation information of other packages, call\n\ncitation(\"[package]\")\n\n\n\n\n\n\n\nReferences\n\nBinder, Martin, Florian Pfisterer, Michel Lang, Lennart Schneider, Lars Kotthoff, and Bernd Bischl. 2021. “mlr3pipelines - Flexible Machine Learning Pipelines in R.” Journal of Machine Learning Research 22 (184): 1–7. https://jmlr.org/papers/v22/21-0281.html.\n\n\nLang, Michel, Martin Binder, Jakob Richter, Patrick Schratz, Florian Pfisterer, Stefan Coors, Quay Au, Giuseppe Casalicchio, Lars Kotthoff, and Bernd Bischl. 2019. “mlr3: A Modern Object-Oriented Machine Learning Framework in R.” Journal of Open Source Software, December. https://doi.org/10.21105/joss.01903.\n\n\nSonabend, Raphael, Franz J Király, Andreas Bender, Bernd Bischl, and Michel Lang. 2021. “mlr3proba: An R Package for Machine Learning in Survival Analysis.” Bioinformatics, February. https://doi.org/10.1093/bioinformatics/btab039." - }, - { - "objectID": "tuning_spaces.html", - "href": "tuning_spaces.html", - "title": "", - "section": "", - "text": "The package mlr3tuningspaces ships with some predefined tuning spaces for hyperparameter optimization. See the respective manual page for the article from which they were extracted.\n\n\n\n\n\n\n\n\n\nLoad a tuning space for the classification tree learner from the Bischl et al. (2021) article.\n\nlibrary(mlr3verse)\n\n# load learner and set search space\nlearner = lts(lrn(\"classif.rpart\"))\n\n# retrieve task\ntask = tsk(\"pima\")\n\n# load tuner and set batch size\ntuner = tnr(\"random_search\", batch_size = 10)\n\n# hyperparameter tuning on the pima data set\ninstance = tune(\n method = tnr(\"grid_search\", resolution = 5, batch_size = 25),\n task = task,\n learner = learner,\n resampling = rsmp(\"holdout\"),\n measure = msr(\"classif.ce\"),\n)\n\n# best performing hyperparameter configuration\ninstance$result\n\n minsplit minbucket cp learner_param_vals x_domain classif.ce\n1: 4.859812 0 -9.21034 0.2070312\n\n# fit final model on complete data set\nlearner$param_set$values = instance$result_learner_param_vals\nlearner$train(task)\n\nprint(learner)\n\n: Classification Tree\n* Model: rpart\n* Parameters: xval=0, minsplit=128, minbucket=1, cp=0.0001\n* Packages: mlr3, rpart\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, multiclass, selected_features,\n twoclass, weights" - }, - { - "objectID": "resources.html", - "href": "resources.html", - "title": "Resources", - "section": "", - "text": "mlr3book\nmlr3gallery\n{R6} class system introduction\n{future} package - parallelization backend" - }, - { - "objectID": "resources.html#videos", - "href": "resources.html#videos", - "title": "Resources", - "section": "Videos", - "text": "Videos\n\nuseR2019 talk on mlr3\nuseR2019 talk on mlr3pipelines and mlr3tuning\nuseR2020 tutorial on mlr3, mlr3tuning and mlr3pipelines\nRecorded talk about mlr3spatiotempcv and mlr3spatial at OpenDataScience Europe Conference 2021 in Wageningen, NL" - }, - { - "objectID": "resources.html#cheatsheets", - "href": "resources.html#cheatsheets", - "title": "Resources", - "section": "Cheatsheets", - "text": "Cheatsheets\n\nOverview of cheatsheets\nmlr3\nmlr3tuning\nmlr3pipelines" - }, - { - "objectID": "resources.html#courseslectures", - "href": "resources.html#courseslectures", - "title": "Resources", - "section": "Courses/lectures", - "text": "Courses/lectures\n\nThe course Introduction to Machine learning (I2ML) is a free and open flipped classroom course on the basics of machine learning. mlr3 is used in the demos and exercises." - }, - { - "objectID": "resources.html#peer-reviewed-articles", - "href": "resources.html#peer-reviewed-articles", - "title": "Resources", - "section": "Peer-reviewed Articles", - "text": "Peer-reviewed Articles\nA more scientific view on our the packages and the packages we depend on.\n\nLang et al. (2019): about the base package mlr3\nBinder et al. (2021): building machine learning pipelines with mlr3pipelines\nSonabend et al. (2021): probabilistic regression with mlr3proba (including survival analysis)\nBengtsson (2021): the parallelization framework package future we build upon\nLang (2017): package checkmate for argument checking and defensive programming\nLang, Bischl, and Surmann (2017): parallelization framework batchtools for high-performance computing clusters, used via future or mlr3batchmark" - }, - { - "objectID": "fselectors.html", - "href": "fselectors.html", - "title": "", - "section": "", - "text": "Feature selection wrappers can be found in the mlr3fselect packages. The goal is to find the best subset of features with respect to a performance measure in an iterative fashion.\n\n\n\n\n\n\n\n\n\nRun a sequential feature selection on the Pima Indian Diabetes data set.\n\nlibrary(mlr3verse)\n\n# retrieve task\ntask = tsk(\"pima\")\n\n# load learner\nlearner = lrn(\"classif.rpart\")\n\n# feature selection on the pima indians diabetes data set\ninstance = fselect(\n method = fs(\"sequential\"),\n task = task,\n learner = learner,\n resampling = rsmp(\"holdout\"),\n measure = msr(\"classif.ce\")\n)\n\n# best performing feature subset\ninstance$result\n\n age glucose insulin mass pedigree pregnant pressure triceps\n1: TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE\n features classif.ce\n1: age,glucose,insulin,mass,pregnant,pressure,... 0.2382812\n\n# subset the task and fit the final model\ntask$select(instance$result_feature_set)\nlearner$train(task)\n\nprint(learner)\n\n: Classification Tree\n* Model: rpart\n* Parameters: xval=0\n* Packages: mlr3, rpart\n* Predict Types: [response], prob\n* Feature Types: logical, integer, numeric, factor, ordered\n* Properties: importance, missings, multiclass, selected_features,\n twoclass, weights" - }, - { - "objectID": "blog.html", - "href": "blog.html", - "title": "Blog", - "section": "", - "text": "mlr3 package updates - q1/2022\n\n\n\n\n\nQuarterly mlr3 package updates. This posts gives an overview by listing the recent release notes of mlr3 packages from the last quarter.\n\n\n\n\n\n\nApr 25, 2022\n\n\nSebastian Fischer\n\n\n\n\n\n\n \n\n\n\n\nGoogle Summer of Code and mlr3\n\n\n\n\n\nOur Org is currently thinking about participating in GSOC 2022 again.\n\n\n\n\n\n\nFeb 3, 2022\n\n\nFlorian Pfisterer\n\n\n\n\n\n\n \n\n\n\n\nmlr3 package Updates - Q4/2021\n\n\n\n\n\nThis posts gives an overview by listing the recent release notes of mlr3 packages from the last quarter.\n\n\n\n\n\n\nJan 3, 2022\n\n\nPatrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nAnnouncing mlr3spatial\n\n\n\n\n\nWe are happy to announce that mlr3spatial has been released on CRAN in November 2021. mlr3spatial simplifies the handling of spatial objects in the mlr3 ecosystem. Before mlr3spatial, the user had to extract tabular data from spatial objects to train a model or predict spatial data.\n\n\n\n\n\n\nDec 1, 2021\n\n\nMarc Becker, Patrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nmlr Workshop 2021 Recap\n\n\n\n\n\nThis blog post is a recap of the mlr-org workshop 2021 which took place from the 28th of September until the 10th of October.\n\n\n\n\n\n\nOct 7, 2021\n\n\nPatrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nmlr3 Package Updates - Q3/2021\n\n\n\n\n\nThis posts gives an overview by listing the recent release notes of mlr3 packages from the last quarter.\n\n\n\n\n\n\nSep 29, 2021\n\n\nPatrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nThe Cross-Validation - Train/Predict Misunderstanding\n\n\n\n\n\nOver the past years I’ve seen multiple posts on Stackoverflow and our GitHub issues which suffer from a conceptual misunderstanding: cross-validation (CV) vs. train/predict.\n\n\n\n\n\n\nDec 20, 2020\n\n\nPatrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nAnnouncing mlr3spatiotempcv\n\n\n\n\n\nWe are happy to announce that mlr3spatial has been released on CRAN in November 2020.\n\n\n\n\n\n\nNov 12, 2020\n\n\nPatrick Schratz\n\n\n\n\n\n\n \n\n\n\n\nIntroducing mlr3cluster: Cluster Analysis Package\n\n\n\n\n\nTired of learning to use multiple packages to access clustering algorithms?\n\n\n\n\n\n\nAug 26, 2020\n\n\nDamir Pulatov\n\n\n\n\n\n\n \n\n\n\n\nuseR 2020 Tutorial on mlr3, mlr3tuning and mlr3pipelines\n\n\n\n\n\nInvitation to useR tutorial on mlr3, mlr3tuning and pipelines.\n\n\n\n\n\n\nAug 4, 2020\n\n\nMarc Becker\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWebinar on mlr3pipelines\n\n\n\n\n\nAnnouncing the WhyR Webinar on mlr3pipelines\n\n\n\n\n\n\nMay 20, 2020\n\n\n\n\n\n\n \n\n\n\n\nExplainable machine learning with mlr3 and DALEX\n\n\n\n\n\nNew mlr3 book chapter about interpretable machine learning using DALEX\n\n\n\n\n\n\nMar 30, 2020\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr3 tutorial on useR!2020muc\n\n\n\n\n\nAnnouncing a mlr3 tutorial at userR!2020\n\n\n\n\n\n\nMar 6, 2020\n\n\n\n\n\n\n \n\n\n\n\nmlr wins Open Source Machine Learning Software Award\n\n\n\n\n\nmlr wins Open Source Machine Learning Project award\n\n\n\n\n\n\nDec 21, 2019\n\n\n\n\n\n\n \n\n\n\n\nIntroducing mlrPlayground\n\n\n\n\n\nExplore mlr interactively using mlrPlayground\n\n\n\n\n\n\nAug 12, 2019\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr-2.15.0\n\n\n\n\n\nSummary of changes in mlr-2.15.0\n\n\n\n\n\n\nAug 6, 2019\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr3-0.1.0\n\n\n\n\n\nInitial release of mlr3\n\n\n\n\n\n\nJul 31, 2019\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr + drake: Reproducible machine-learning workflow management\n\n\n\n\n\nUsing mlr and drake\n\n\n\n\n\n\nMay 6, 2019\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr-2.14.0\n\n\n\n\n\nThe new mlr-2.14.0 and announcement of mlr3\n\n\n\n\n\n\nApr 18, 2019\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nVisualization of spatial cross-validation partitioning\n\n\n\n\n\nVisualizing spatial cross-validation in mlr\n\n\n\n\n\n\nJul 25, 2018\n\n\n\n\n\n\n \n\n\n\n\nWhy R Conference\n\n\n\n\n\nSummary of the mlr tutorial at WhyR in Wroclaw\n\n\n\n\n\n\nJul 5, 2018\n\n\n\n\n\n\n \n\n\n\n\nInterpretable Machine Learning with iml and mlr\n\n\n\n\n\nTutorial on the iml R package\n\n\n\n\n\n\nFeb 28, 2018\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTraining Courses for mlr: Machine Learning in R\n\n\n\n\n\nAnnouncing mlr training courses\n\n\n\n\n\n\nFeb 28, 2018\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nStepwise Bayesian Optimization with mlrMBO\n\n\n\n\n\nTutorial on stepwise Bayesian Optimization with mlrMBO\n\n\n\n\n\n\nJan 10, 2018\n\n\n\n\n\n\n \n\n\n\n\nTeam Rtus wins Munich Re Datathon with mlr\n\n\n\n\n\nWinning the RE Datathon with mlr\n\n\n\n\n\n\nDec 14, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nOpenML Workshop 2017\n\n\n\n\n\nAnnouncing the OpenML workshop\n\n\n\n\n\n\nSep 1, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nParameter tuning with mlrHyperopt\n\n\n\n\n\nTutorial on mlrHyperopt\n\n\n\n\n\n\nJul 19, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nshinyMlr\n\n\n\n\n\nTutorial series on how to use shinyMlr\n\n\n\n\n\n\nMay 16, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nMost Popular Learners in mlr\n\n\n\n\n\nAn attempt to asses the popularity of mlr learners\n\n\n\n\n\n\nMar 30, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nMultilabel Classification with mlr\n\n\n\n\n\nIntroducting to multilabel classification in mlr\n\n\n\n\n\n\nMar 28, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nNew mlr Logo\n\n\n\n\n\nVoting for a new mlr logo\n\n\n\n\n\n\nMar 24, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nUse mlrMBO to optimize via command line\n\n\n\n\n\nTutorial on using mlrMBO from the command line\n\n\n\n\n\n\nMar 22, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFirst release of mlrMBO - the toolbox for Bayesian Block Box Optimization\n\n\n\n\n\nA short description of the post.\n\n\n\n\n\n\nMar 13, 2017\n\n\nJakob Richter\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBeing successful on Kaggle using mlr\n\n\n\n\n\nTutorial on how to be successful on Kaggle using mlr\n\n\n\n\n\n\nMar 9, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nOpenML tutorial at useR!2017 Brussels\n\n\n\n\n\nAnnouncing an OpenML presentation at useR\n\n\n\n\n\n\nMar 2, 2017\n\n\n\n\n\n\n \n\n\n\n\nmlr Workshop 2017\n\n\n\n\n\nAnnouncing the mlr workshop\n\n\n\n\n\n\nFeb 13, 2017\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nGSOC\n\n\n\n\n\nAnnouncing the mlr GSOC project: preprocessing pipelines\n\n\n\n\n\n\nFeb 13, 2017\n\n\nJanek Thomas\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmlr 2.10\n\n\n\n\n\nSummary of the changes in the new mlr 2.10 version\n\n\n\n\n\n\nFeb 13, 2017\n\n\n\n\n\n\n \n\n\n\n\nPaper published: mlr - Machine Learning in R\n\n\n\n\n\nAnnouncing the publication of the mlr paper.\n\n\n\n\n\n\nOct 20, 2016\n\n\n\n\n\n\n \n\n\n\n\nmlr loves OpenML\n\n\n\n\n\nConnecting mlr with OpenML\n\n\n\n\n\n\nSep 9, 2016\n\n\nHeidi Seibold\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nHow to win a drone in 20 lines of R code\n\n\n\n\n\nShowcasing model tuning using mlr.\n\n\n\n\n\n\nAug 23, 2016\n\n\nJanek Thomas\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExploring and Understanding Hyperparameter Tuning\n\n\n\n\n\nHyperparameter optimization in mlr\n\n\n\n\n\n\nAug 21, 2016\n\n\nMason Gallo\n\n\n\n\n\n\n \n\n\n\n\nResult of the mlr summer workshop in Palermo\n\n\n\n\n\nA summary of the results of the mlr summer workshop in Palermo.\n\n\n\n\n\n\nAug 15, 2016\n\n\nJanek Thomas\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExploring Learner Predictions with Partial Dependence and Functional ANOVA\n\n\n\n\n\nUsing partial dependency plots in mlr.\n\n\n\n\n\n\nAug 11, 2016\n\n\nZachary Jones\n\n\n\n\n\n\n \n\n\n\n\nBenchmarking mlr learners on OpenML\n\n\n\n\n\nBenchmarking mlr learners on the OpenML platform.\n\n\n\n\n\n\nAug 11, 2016\n\n\nFlorian Pfisterer\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nVisualization of predictions\n\n\n\n\n\nTutorial on the visualization of predictions using mlr.\n\n\n\n\n\n\nJul 28, 2015\n\n\n\n\n\n\nNo matching items" - }, - { - "objectID": "graphs.html", - "href": "graphs.html", - "title": "", - "section": "", - "text": "Graphs\nGraphs are predefined arrangements of PipeOp objects from the mlr3pipelines package. The goal is to simplify some popular operations which usually consist of multiple steps." - }, - { - "objectID": "documentation-listings/gallery.html", - "href": "documentation-listings/gallery.html", - "title": "Gallery", - "section": "", - "text": "foo" - } -] \ No newline at end of file diff --git a/mlr-org/_site/site_libs/jquery-3.6.0/jquery-3.6.0.js b/mlr-org/_site/site_libs/jquery-3.6.0/jquery-3.6.0.js deleted file mode 100644 index fc6c299b7..000000000 --- a/mlr-org/_site/site_libs/jquery-3.6.0/jquery-3.6.0.js +++ /dev/null @@ -1,10881 +0,0 @@ -/*! - * jQuery JavaScript Library v3.6.0 - * https://jquery.com/ - * - * Includes Sizzle.js - * https://sizzlejs.com/ - * - * Copyright OpenJS Foundation and other contributors - * Released under the MIT license - * https://jquery.org/license - * - * Date: 2021-03-02T17:08Z - */ -( function( global, factory ) { - - "use strict"; - - if ( typeof module === "object" && typeof module.exports === "object" ) { - - // For CommonJS and CommonJS-like environments where a proper `window` - // is present, execute the factory and get jQuery. - // For environments that do not have a `window` with a `document` - // (such as Node.js), expose a factory as module.exports. - // This accentuates the need for the creation of a real `window`. - // e.g. var jQuery = require("jquery")(window); - // See ticket #14549 for more info. - module.exports = global.document ? - factory( global, true ) : - function( w ) { - if ( !w.document ) { - throw new Error( "jQuery requires a window with a document" ); - } - return factory( w ); - }; - } else { - factory( global ); - } - -// Pass this if window is not defined yet -} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { - -// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 -// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode -// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common -// enough that all such attempts are guarded in a try block. -"use strict"; - -var arr = []; - -var getProto = Object.getPrototypeOf; - -var slice = arr.slice; - -var flat = arr.flat ? function( array ) { - return arr.flat.call( array ); -} : function( array ) { - return arr.concat.apply( [], array ); -}; - - -var push = arr.push; - -var indexOf = arr.indexOf; - -var class2type = {}; - -var toString = class2type.toString; - -var hasOwn = class2type.hasOwnProperty; - -var fnToString = hasOwn.toString; - -var ObjectFunctionString = fnToString.call( Object ); - -var support = {}; - -var isFunction = function isFunction( obj ) { - - // Support: Chrome <=57, Firefox <=52 - // In some browsers, typeof returns "function" for HTML elements - // (i.e., `typeof document.createElement( "object" ) === "function"`). - // We don't want to classify *any* DOM node as a function. - // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 - // Plus for old WebKit, typeof returns "function" for HTML collections - // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) - return typeof obj === "function" && typeof obj.nodeType !== "number" && - typeof obj.item !== "function"; - }; - - -var isWindow = function isWindow( obj ) { - return obj != null && obj === obj.window; - }; - - -var document = window.document; - - - - var preservedScriptAttributes = { - type: true, - src: true, - nonce: true, - noModule: true - }; - - function DOMEval( code, node, doc ) { - doc = doc || document; - - var i, val, - script = doc.createElement( "script" ); - - script.text = code; - if ( node ) { - for ( i in preservedScriptAttributes ) { - - // Support: Firefox 64+, Edge 18+ - // Some browsers don't support the "nonce" property on scripts. - // On the other hand, just using `getAttribute` is not enough as - // the `nonce` attribute is reset to an empty string whenever it - // becomes browsing-context connected. - // See https://github.com/whatwg/html/issues/2369 - // See https://html.spec.whatwg.org/#nonce-attributes - // The `node.getAttribute` check was added for the sake of - // `jQuery.globalEval` so that it can fake a nonce-containing node - // via an object. - val = node[ i ] || node.getAttribute && node.getAttribute( i ); - if ( val ) { - script.setAttribute( i, val ); - } - } - } - doc.head.appendChild( script ).parentNode.removeChild( script ); - } - - -function toType( obj ) { - if ( obj == null ) { - return obj + ""; - } - - // Support: Android <=2.3 only (functionish RegExp) - return typeof obj === "object" || typeof obj === "function" ? - class2type[ toString.call( obj ) ] || "object" : - typeof obj; -} -/* global Symbol */ -// Defining this global in .eslintrc.json would create a danger of using the global -// unguarded in another place, it seems safer to define global only for this module - - - -var - version = "3.6.0", - - // Define a local copy of jQuery - jQuery = function( selector, context ) { - - // The jQuery object is actually just the init constructor 'enhanced' - // Need init if jQuery is called (just allow error to be thrown if not included) - return new jQuery.fn.init( selector, context ); - }; - -jQuery.fn = jQuery.prototype = { - - // The current version of jQuery being used - jquery: version, - - constructor: jQuery, - - // The default length of a jQuery object is 0 - length: 0, - - toArray: function() { - return slice.call( this ); - }, - - // Get the Nth element in the matched element set OR - // Get the whole matched element set as a clean array - get: function( num ) { - - // Return all the elements in a clean array - if ( num == null ) { - return slice.call( this ); - } - - // Return just the one element from the set - return num < 0 ? this[ num + this.length ] : this[ num ]; - }, - - // Take an array of elements and push it onto the stack - // (returning the new matched element set) - pushStack: function( elems ) { - - // Build a new jQuery matched element set - var ret = jQuery.merge( this.constructor(), elems ); - - // Add the old object onto the stack (as a reference) - ret.prevObject = this; - - // Return the newly-formed element set - return ret; - }, - - // Execute a callback for every element in the matched set. - each: function( callback ) { - return jQuery.each( this, callback ); - }, - - map: function( callback ) { - return this.pushStack( jQuery.map( this, function( elem, i ) { - return callback.call( elem, i, elem ); - } ) ); - }, - - slice: function() { - return this.pushStack( slice.apply( this, arguments ) ); - }, - - first: function() { - return this.eq( 0 ); - }, - - last: function() { - return this.eq( -1 ); - }, - - even: function() { - return this.pushStack( jQuery.grep( this, function( _elem, i ) { - return ( i + 1 ) % 2; - } ) ); - }, - - odd: function() { - return this.pushStack( jQuery.grep( this, function( _elem, i ) { - return i % 2; - } ) ); - }, - - eq: function( i ) { - var len = this.length, - j = +i + ( i < 0 ? len : 0 ); - return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); - }, - - end: function() { - return this.prevObject || this.constructor(); - }, - - // For internal use only. - // Behaves like an Array's method, not like a jQuery method. - push: push, - sort: arr.sort, - splice: arr.splice -}; - -jQuery.extend = jQuery.fn.extend = function() { - var options, name, src, copy, copyIsArray, clone, - target = arguments[ 0 ] || {}, - i = 1, - length = arguments.length, - deep = false; - - // Handle a deep copy situation - if ( typeof target === "boolean" ) { - deep = target; - - // Skip the boolean and the target - target = arguments[ i ] || {}; - i++; - } - - // Handle case when target is a string or something (possible in deep copy) - if ( typeof target !== "object" && !isFunction( target ) ) { - target = {}; - } - - // Extend jQuery itself if only one argument is passed - if ( i === length ) { - target = this; - i--; - } - - for ( ; i < length; i++ ) { - - // Only deal with non-null/undefined values - if ( ( options = arguments[ i ] ) != null ) { - - // Extend the base object - for ( name in options ) { - copy = options[ name ]; - - // Prevent Object.prototype pollution - // Prevent never-ending loop - if ( name === "__proto__" || target === copy ) { - continue; - } - - // Recurse if we're merging plain objects or arrays - if ( deep && copy && ( jQuery.isPlainObject( copy ) || - ( copyIsArray = Array.isArray( copy ) ) ) ) { - src = target[ name ]; - - // Ensure proper type for the source value - if ( copyIsArray && !Array.isArray( src ) ) { - clone = []; - } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { - clone = {}; - } else { - clone = src; - } - copyIsArray = false; - - // Never move original objects, clone them - target[ name ] = jQuery.extend( deep, clone, copy ); - - // Don't bring in undefined values - } else if ( copy !== undefined ) { - target[ name ] = copy; - } - } - } - } - - // Return the modified object - return target; -}; - -jQuery.extend( { - - // Unique for each copy of jQuery on the page - expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), - - // Assume jQuery is ready without the ready module - isReady: true, - - error: function( msg ) { - throw new Error( msg ); - }, - - noop: function() {}, - - isPlainObject: function( obj ) { - var proto, Ctor; - - // Detect obvious negatives - // Use toString instead of jQuery.type to catch host objects - if ( !obj || toString.call( obj ) !== "[object Object]" ) { - return false; - } - - proto = getProto( obj ); - - // Objects with no prototype (e.g., `Object.create( null )`) are plain - if ( !proto ) { - return true; - } - - // Objects with prototype are plain iff they were constructed by a global Object function - Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; - return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; - }, - - isEmptyObject: function( obj ) { - var name; - - for ( name in obj ) { - return false; - } - return true; - }, - - // Evaluates a script in a provided context; falls back to the global one - // if not specified. - globalEval: function( code, options, doc ) { - DOMEval( code, { nonce: options && options.nonce }, doc ); - }, - - each: function( obj, callback ) { - var length, i = 0; - - if ( isArrayLike( obj ) ) { - length = obj.length; - for ( ; i < length; i++ ) { - if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { - break; - } - } - } else { - for ( i in obj ) { - if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { - break; - } - } - } - - return obj; - }, - - // results is for internal usage only - makeArray: function( arr, results ) { - var ret = results || []; - - if ( arr != null ) { - if ( isArrayLike( Object( arr ) ) ) { - jQuery.merge( ret, - typeof arr === "string" ? - [ arr ] : arr - ); - } else { - push.call( ret, arr ); - } - } - - return ret; - }, - - inArray: function( elem, arr, i ) { - return arr == null ? -1 : indexOf.call( arr, elem, i ); - }, - - // Support: Android <=4.0 only, PhantomJS 1 only - // push.apply(_, arraylike) throws on ancient WebKit - merge: function( first, second ) { - var len = +second.length, - j = 0, - i = first.length; - - for ( ; j < len; j++ ) { - first[ i++ ] = second[ j ]; - } - - first.length = i; - - return first; - }, - - grep: function( elems, callback, invert ) { - var callbackInverse, - matches = [], - i = 0, - length = elems.length, - callbackExpect = !invert; - - // Go through the array, only saving the items - // that pass the validator function - for ( ; i < length; i++ ) { - callbackInverse = !callback( elems[ i ], i ); - if ( callbackInverse !== callbackExpect ) { - matches.push( elems[ i ] ); - } - } - - return matches; - }, - - // arg is for internal usage only - map: function( elems, callback, arg ) { - var length, value, - i = 0, - ret = []; - - // Go through the array, translating each of the items to their new values - if ( isArrayLike( elems ) ) { - length = elems.length; - for ( ; i < length; i++ ) { - value = callback( elems[ i ], i, arg ); - - if ( value != null ) { - ret.push( value ); - } - } - - // Go through every key on the object, - } else { - for ( i in elems ) { - value = callback( elems[ i ], i, arg ); - - if ( value != null ) { - ret.push( value ); - } - } - } - - // Flatten any nested arrays - return flat( ret ); - }, - - // A global GUID counter for objects - guid: 1, - - // jQuery.support is not used in Core but other projects attach their - // properties to it so it needs to exist. - support: support -} ); - -if ( typeof Symbol === "function" ) { - jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; -} - -// Populate the class2type map -jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), - function( _i, name ) { - class2type[ "[object " + name + "]" ] = name.toLowerCase(); - } ); - -function isArrayLike( obj ) { - - // Support: real iOS 8.2 only (not reproducible in simulator) - // `in` check used to prevent JIT error (gh-2145) - // hasOwn isn't used here due to false negatives - // regarding Nodelist length in IE - var length = !!obj && "length" in obj && obj.length, - type = toType( obj ); - - if ( isFunction( obj ) || isWindow( obj ) ) { - return false; - } - - return type === "array" || length === 0 || - typeof length === "number" && length > 0 && ( length - 1 ) in obj; -} -var Sizzle = -/*! - * Sizzle CSS Selector Engine v2.3.6 - * https://sizzlejs.com/ - * - * Copyright JS Foundation and other contributors - * Released under the MIT license - * https://js.foundation/ - * - * Date: 2021-02-16 - */ -( function( window ) { -var i, - support, - Expr, - getText, - isXML, - tokenize, - compile, - select, - outermostContext, - sortInput, - hasDuplicate, - - // Local document vars - setDocument, - document, - docElem, - documentIsHTML, - rbuggyQSA, - rbuggyMatches, - matches, - contains, - - // Instance-specific data - expando = "sizzle" + 1 * new Date(), - preferredDoc = window.document, - dirruns = 0, - done = 0, - classCache = createCache(), - tokenCache = createCache(), - compilerCache = createCache(), - nonnativeSelectorCache = createCache(), - sortOrder = function( a, b ) { - if ( a === b ) { - hasDuplicate = true; - } - return 0; - }, - - // Instance methods - hasOwn = ( {} ).hasOwnProperty, - arr = [], - pop = arr.pop, - pushNative = arr.push, - push = arr.push, - slice = arr.slice, - - // Use a stripped-down indexOf as it's faster than native - // https://jsperf.com/thor-indexof-vs-for/5 - indexOf = function( list, elem ) { - var i = 0, - len = list.length; - for ( ; i < len; i++ ) { - if ( list[ i ] === elem ) { - return i; - } - } - return -1; - }, - - booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + - "ismap|loop|multiple|open|readonly|required|scoped", - - // Regular expressions - - // http://www.w3.org/TR/css3-selectors/#whitespace - whitespace = "[\\x20\\t\\r\\n\\f]", - - // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram - identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + - "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", - - // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors - attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + - - // Operator (capture 2) - "*([*^$|!~]?=)" + whitespace + - - // "Attribute values must be CSS identifiers [capture 5] - // or strings [capture 3 or capture 4]" - "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + - whitespace + "*\\]", - - pseudos = ":(" + identifier + ")(?:\\((" + - - // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: - // 1. quoted (capture 3; capture 4 or capture 5) - "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + - - // 2. simple (capture 6) - "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + - - // 3. anything else (capture 2) - ".*" + - ")\\)|)", - - // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter - rwhitespace = new RegExp( whitespace + "+", "g" ), - rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + - whitespace + "+$", "g" ), - - rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), - rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + - "*" ), - rdescend = new RegExp( whitespace + "|>" ), - - rpseudo = new RegExp( pseudos ), - ridentifier = new RegExp( "^" + identifier + "$" ), - - matchExpr = { - "ID": new RegExp( "^#(" + identifier + ")" ), - "CLASS": new RegExp( "^\\.(" + identifier + ")" ), - "TAG": new RegExp( "^(" + identifier + "|[*])" ), - "ATTR": new RegExp( "^" + attributes ), - "PSEUDO": new RegExp( "^" + pseudos ), - "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + - whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + - whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), - "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), - - // For use in libraries implementing .is() - // We use this for POS matching in `select` - "needsContext": new RegExp( "^" + whitespace + - "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + - "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) - }, - - rhtml = /HTML$/i, - rinputs = /^(?:input|select|textarea|button)$/i, - rheader = /^h\d$/i, - - rnative = /^[^{]+\{\s*\[native \w/, - - // Easily-parseable/retrievable ID or TAG or CLASS selectors - rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, - - rsibling = /[+~]/, - - // CSS escapes - // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters - runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), - funescape = function( escape, nonHex ) { - var high = "0x" + escape.slice( 1 ) - 0x10000; - - return nonHex ? - - // Strip the backslash prefix from a non-hex escape sequence - nonHex : - - // Replace a hexadecimal escape sequence with the encoded Unicode code point - // Support: IE <=11+ - // For values outside the Basic Multilingual Plane (BMP), manually construct a - // surrogate pair - high < 0 ? - String.fromCharCode( high + 0x10000 ) : - String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); - }, - - // CSS string/identifier serialization - // https://drafts.csswg.org/cssom/#common-serializing-idioms - rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, - fcssescape = function( ch, asCodePoint ) { - if ( asCodePoint ) { - - // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER - if ( ch === "\0" ) { - return "\uFFFD"; - } - - // Control characters and (dependent upon position) numbers get escaped as code points - return ch.slice( 0, -1 ) + "\\" + - ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; - } - - // Other potentially-special ASCII characters get backslash-escaped - return "\\" + ch; - }, - - // Used for iframes - // See setDocument() - // Removing the function wrapper causes a "Permission Denied" - // error in IE - unloadHandler = function() { - setDocument(); - }, - - inDisabledFieldset = addCombinator( - function( elem ) { - return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; - }, - { dir: "parentNode", next: "legend" } - ); - -// Optimize for push.apply( _, NodeList ) -try { - push.apply( - ( arr = slice.call( preferredDoc.childNodes ) ), - preferredDoc.childNodes - ); - - // Support: Android<4.0 - // Detect silently failing push.apply - // eslint-disable-next-line no-unused-expressions - arr[ preferredDoc.childNodes.length ].nodeType; -} catch ( e ) { - push = { apply: arr.length ? - - // Leverage slice if possible - function( target, els ) { - pushNative.apply( target, slice.call( els ) ); - } : - - // Support: IE<9 - // Otherwise append directly - function( target, els ) { - var j = target.length, - i = 0; - - // Can't trust NodeList.length - while ( ( target[ j++ ] = els[ i++ ] ) ) {} - target.length = j - 1; - } - }; -} - -function Sizzle( selector, context, results, seed ) { - var m, i, elem, nid, match, groups, newSelector, - newContext = context && context.ownerDocument, - - // nodeType defaults to 9, since context defaults to document - nodeType = context ? context.nodeType : 9; - - results = results || []; - - // Return early from calls with invalid selector or context - if ( typeof selector !== "string" || !selector || - nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { - - return results; - } - - // Try to shortcut find operations (as opposed to filters) in HTML documents - if ( !seed ) { - setDocument( context ); - context = context || document; - - if ( documentIsHTML ) { - - // If the selector is sufficiently simple, try using a "get*By*" DOM method - // (excepting DocumentFragment context, where the methods don't exist) - if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { - - // ID selector - if ( ( m = match[ 1 ] ) ) { - - // Document context - if ( nodeType === 9 ) { - if ( ( elem = context.getElementById( m ) ) ) { - - // Support: IE, Opera, Webkit - // TODO: identify versions - // getElementById can match elements by name instead of ID - if ( elem.id === m ) { - results.push( elem ); - return results; - } - } else { - return results; - } - - // Element context - } else { - - // Support: IE, Opera, Webkit - // TODO: identify versions - // getElementById can match elements by name instead of ID - if ( newContext && ( elem = newContext.getElementById( m ) ) && - contains( context, elem ) && - elem.id === m ) { - - results.push( elem ); - return results; - } - } - - // Type selector - } else if ( match[ 2 ] ) { - push.apply( results, context.getElementsByTagName( selector ) ); - return results; - - // Class selector - } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && - context.getElementsByClassName ) { - - push.apply( results, context.getElementsByClassName( m ) ); - return results; - } - } - - // Take advantage of querySelectorAll - if ( support.qsa && - !nonnativeSelectorCache[ selector + " " ] && - ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && - - // Support: IE 8 only - // Exclude object elements - ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { - - newSelector = selector; - newContext = context; - - // qSA considers elements outside a scoping root when evaluating child or - // descendant combinators, which is not what we want. - // In such cases, we work around the behavior by prefixing every selector in the - // list with an ID selector referencing the scope context. - // The technique has to be used as well when a leading combinator is used - // as such selectors are not recognized by querySelectorAll. - // Thanks to Andrew Dupont for this technique. - if ( nodeType === 1 && - ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { - - // Expand context for sibling selectors - newContext = rsibling.test( selector ) && testContext( context.parentNode ) || - context; - - // We can use :scope instead of the ID hack if the browser - // supports it & if we're not changing the context. - if ( newContext !== context || !support.scope ) { - - // Capture the context ID, setting it first if necessary - if ( ( nid = context.getAttribute( "id" ) ) ) { - nid = nid.replace( rcssescape, fcssescape ); - } else { - context.setAttribute( "id", ( nid = expando ) ); - } - } - - // Prefix every selector in the list - groups = tokenize( selector ); - i = groups.length; - while ( i-- ) { - groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + - toSelector( groups[ i ] ); - } - newSelector = groups.join( "," ); - } - - try { - push.apply( results, - newContext.querySelectorAll( newSelector ) - ); - return results; - } catch ( qsaError ) { - nonnativeSelectorCache( selector, true ); - } finally { - if ( nid === expando ) { - context.removeAttribute( "id" ); - } - } - } - } - } - - // All others - return select( selector.replace( rtrim, "$1" ), context, results, seed ); -} - -/** - * Create key-value caches of limited size - * @returns {function(string, object)} Returns the Object data after storing it on itself with - * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) - * deleting the oldest entry - */ -function createCache() { - var keys = []; - - function cache( key, value ) { - - // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) - if ( keys.push( key + " " ) > Expr.cacheLength ) { - - // Only keep the most recent entries - delete cache[ keys.shift() ]; - } - return ( cache[ key + " " ] = value ); - } - return cache; -} - -/** - * Mark a function for special use by Sizzle - * @param {Function} fn The function to mark - */ -function markFunction( fn ) { - fn[ expando ] = true; - return fn; -} - -/** - * Support testing using an element - * @param {Function} fn Passed the created element and returns a boolean result - */ -function assert( fn ) { - var el = document.createElement( "fieldset" ); - - try { - return !!fn( el ); - } catch ( e ) { - return false; - } finally { - - // Remove from its parent by default - if ( el.parentNode ) { - el.parentNode.removeChild( el ); - } - - // release memory in IE - el = null; - } -} - -/** - * Adds the same handler for all of the specified attrs - * @param {String} attrs Pipe-separated list of attributes - * @param {Function} handler The method that will be applied - */ -function addHandle( attrs, handler ) { - var arr = attrs.split( "|" ), - i = arr.length; - - while ( i-- ) { - Expr.attrHandle[ arr[ i ] ] = handler; - } -} - -/** - * Checks document order of two siblings - * @param {Element} a - * @param {Element} b - * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b - */ -function siblingCheck( a, b ) { - var cur = b && a, - diff = cur && a.nodeType === 1 && b.nodeType === 1 && - a.sourceIndex - b.sourceIndex; - - // Use IE sourceIndex if available on both nodes - if ( diff ) { - return diff; - } - - // Check if b follows a - if ( cur ) { - while ( ( cur = cur.nextSibling ) ) { - if ( cur === b ) { - return -1; - } - } - } - - return a ? 1 : -1; -} - -/** - * Returns a function to use in pseudos for input types - * @param {String} type - */ -function createInputPseudo( type ) { - return function( elem ) { - var name = elem.nodeName.toLowerCase(); - return name === "input" && elem.type === type; - }; -} - -/** - * Returns a function to use in pseudos for buttons - * @param {String} type - */ -function createButtonPseudo( type ) { - return function( elem ) { - var name = elem.nodeName.toLowerCase(); - return ( name === "input" || name === "button" ) && elem.type === type; - }; -} - -/** - * Returns a function to use in pseudos for :enabled/:disabled - * @param {Boolean} disabled true for :disabled; false for :enabled - */ -function createDisabledPseudo( disabled ) { - - // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable - return function( elem ) { - - // Only certain elements can match :enabled or :disabled - // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled - // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled - if ( "form" in elem ) { - - // Check for inherited disabledness on relevant non-disabled elements: - // * listed form-associated elements in a disabled fieldset - // https://html.spec.whatwg.org/multipage/forms.html#category-listed - // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled - // * option elements in a disabled optgroup - // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled - // All such elements have a "form" property. - if ( elem.parentNode && elem.disabled === false ) { - - // Option elements defer to a parent optgroup if present - if ( "label" in elem ) { - if ( "label" in elem.parentNode ) { - return elem.parentNode.disabled === disabled; - } else { - return elem.disabled === disabled; - } - } - - // Support: IE 6 - 11 - // Use the isDisabled shortcut property to check for disabled fieldset ancestors - return elem.isDisabled === disabled || - - // Where there is no isDisabled, check manually - /* jshint -W018 */ - elem.isDisabled !== !disabled && - inDisabledFieldset( elem ) === disabled; - } - - return elem.disabled === disabled; - - // Try to winnow out elements that can't be disabled before trusting the disabled property. - // Some victims get caught in our net (label, legend, menu, track), but it shouldn't - // even exist on them, let alone have a boolean value. - } else if ( "label" in elem ) { - return elem.disabled === disabled; - } - - // Remaining elements are neither :enabled nor :disabled - return false; - }; -} - -/** - * Returns a function to use in pseudos for positionals - * @param {Function} fn - */ -function createPositionalPseudo( fn ) { - return markFunction( function( argument ) { - argument = +argument; - return markFunction( function( seed, matches ) { - var j, - matchIndexes = fn( [], seed.length, argument ), - i = matchIndexes.length; - - // Match elements found at the specified indexes - while ( i-- ) { - if ( seed[ ( j = matchIndexes[ i ] ) ] ) { - seed[ j ] = !( matches[ j ] = seed[ j ] ); - } - } - } ); - } ); -} - -/** - * Checks a node for validity as a Sizzle context - * @param {Element|Object=} context - * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value - */ -function testContext( context ) { - return context && typeof context.getElementsByTagName !== "undefined" && context; -} - -// Expose support vars for convenience -support = Sizzle.support = {}; - -/** - * Detects XML nodes - * @param {Element|Object} elem An element or a document - * @returns {Boolean} True iff elem is a non-HTML XML node - */ -isXML = Sizzle.isXML = function( elem ) { - var namespace = elem && elem.namespaceURI, - docElem = elem && ( elem.ownerDocument || elem ).documentElement; - - // Support: IE <=8 - // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes - // https://bugs.jquery.com/ticket/4833 - return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); -}; - -/** - * Sets document-related variables once based on the current document - * @param {Element|Object} [doc] An element or document object to use to set the document - * @returns {Object} Returns the current document - */ -setDocument = Sizzle.setDocument = function( node ) { - var hasCompare, subWindow, - doc = node ? node.ownerDocument || node : preferredDoc; - - // Return early if doc is invalid or already selected - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { - return document; - } - - // Update global variables - document = doc; - docElem = document.documentElement; - documentIsHTML = !isXML( document ); - - // Support: IE 9 - 11+, Edge 12 - 18+ - // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( preferredDoc != document && - ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { - - // Support: IE 11, Edge - if ( subWindow.addEventListener ) { - subWindow.addEventListener( "unload", unloadHandler, false ); - - // Support: IE 9 - 10 only - } else if ( subWindow.attachEvent ) { - subWindow.attachEvent( "onunload", unloadHandler ); - } - } - - // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, - // Safari 4 - 5 only, Opera <=11.6 - 12.x only - // IE/Edge & older browsers don't support the :scope pseudo-class. - // Support: Safari 6.0 only - // Safari 6.0 supports :scope but it's an alias of :root there. - support.scope = assert( function( el ) { - docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); - return typeof el.querySelectorAll !== "undefined" && - !el.querySelectorAll( ":scope fieldset div" ).length; - } ); - - /* Attributes - ---------------------------------------------------------------------- */ - - // Support: IE<8 - // Verify that getAttribute really returns attributes and not properties - // (excepting IE8 booleans) - support.attributes = assert( function( el ) { - el.className = "i"; - return !el.getAttribute( "className" ); - } ); - - /* getElement(s)By* - ---------------------------------------------------------------------- */ - - // Check if getElementsByTagName("*") returns only elements - support.getElementsByTagName = assert( function( el ) { - el.appendChild( document.createComment( "" ) ); - return !el.getElementsByTagName( "*" ).length; - } ); - - // Support: IE<9 - support.getElementsByClassName = rnative.test( document.getElementsByClassName ); - - // Support: IE<10 - // Check if getElementById returns elements by name - // The broken getElementById methods don't pick up programmatically-set names, - // so use a roundabout getElementsByName test - support.getById = assert( function( el ) { - docElem.appendChild( el ).id = expando; - return !document.getElementsByName || !document.getElementsByName( expando ).length; - } ); - - // ID filter and find - if ( support.getById ) { - Expr.filter[ "ID" ] = function( id ) { - var attrId = id.replace( runescape, funescape ); - return function( elem ) { - return elem.getAttribute( "id" ) === attrId; - }; - }; - Expr.find[ "ID" ] = function( id, context ) { - if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { - var elem = context.getElementById( id ); - return elem ? [ elem ] : []; - } - }; - } else { - Expr.filter[ "ID" ] = function( id ) { - var attrId = id.replace( runescape, funescape ); - return function( elem ) { - var node = typeof elem.getAttributeNode !== "undefined" && - elem.getAttributeNode( "id" ); - return node && node.value === attrId; - }; - }; - - // Support: IE 6 - 7 only - // getElementById is not reliable as a find shortcut - Expr.find[ "ID" ] = function( id, context ) { - if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { - var node, i, elems, - elem = context.getElementById( id ); - - if ( elem ) { - - // Verify the id attribute - node = elem.getAttributeNode( "id" ); - if ( node && node.value === id ) { - return [ elem ]; - } - - // Fall back on getElementsByName - elems = context.getElementsByName( id ); - i = 0; - while ( ( elem = elems[ i++ ] ) ) { - node = elem.getAttributeNode( "id" ); - if ( node && node.value === id ) { - return [ elem ]; - } - } - } - - return []; - } - }; - } - - // Tag - Expr.find[ "TAG" ] = support.getElementsByTagName ? - function( tag, context ) { - if ( typeof context.getElementsByTagName !== "undefined" ) { - return context.getElementsByTagName( tag ); - - // DocumentFragment nodes don't have gEBTN - } else if ( support.qsa ) { - return context.querySelectorAll( tag ); - } - } : - - function( tag, context ) { - var elem, - tmp = [], - i = 0, - - // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too - results = context.getElementsByTagName( tag ); - - // Filter out possible comments - if ( tag === "*" ) { - while ( ( elem = results[ i++ ] ) ) { - if ( elem.nodeType === 1 ) { - tmp.push( elem ); - } - } - - return tmp; - } - return results; - }; - - // Class - Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { - if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { - return context.getElementsByClassName( className ); - } - }; - - /* QSA/matchesSelector - ---------------------------------------------------------------------- */ - - // QSA and matchesSelector support - - // matchesSelector(:active) reports false when true (IE9/Opera 11.5) - rbuggyMatches = []; - - // qSa(:focus) reports false when true (Chrome 21) - // We allow this because of a bug in IE8/9 that throws an error - // whenever `document.activeElement` is accessed on an iframe - // So, we allow :focus to pass through QSA all the time to avoid the IE error - // See https://bugs.jquery.com/ticket/13378 - rbuggyQSA = []; - - if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { - - // Build QSA regex - // Regex strategy adopted from Diego Perini - assert( function( el ) { - - var input; - - // Select is set to empty string on purpose - // This is to test IE's treatment of not explicitly - // setting a boolean content attribute, - // since its presence should be enough - // https://bugs.jquery.com/ticket/12359 - docElem.appendChild( el ).innerHTML = "" + - ""; - - // Support: IE8, Opera 11-12.16 - // Nothing should be selected when empty strings follow ^= or $= or *= - // The test attribute must be unknown in Opera but "safe" for WinRT - // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section - if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { - rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); - } - - // Support: IE8 - // Boolean attributes and "value" are not treated correctly - if ( !el.querySelectorAll( "[selected]" ).length ) { - rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); - } - - // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ - if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { - rbuggyQSA.push( "~=" ); - } - - // Support: IE 11+, Edge 15 - 18+ - // IE 11/Edge don't find elements on a `[name='']` query in some cases. - // Adding a temporary attribute to the document before the selection works - // around the issue. - // Interestingly, IE 10 & older don't seem to have the issue. - input = document.createElement( "input" ); - input.setAttribute( "name", "" ); - el.appendChild( input ); - if ( !el.querySelectorAll( "[name='']" ).length ) { - rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + - whitespace + "*(?:''|\"\")" ); - } - - // Webkit/Opera - :checked should return selected option elements - // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked - // IE8 throws error here and will not see later tests - if ( !el.querySelectorAll( ":checked" ).length ) { - rbuggyQSA.push( ":checked" ); - } - - // Support: Safari 8+, iOS 8+ - // https://bugs.webkit.org/show_bug.cgi?id=136851 - // In-page `selector#id sibling-combinator selector` fails - if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { - rbuggyQSA.push( ".#.+[+~]" ); - } - - // Support: Firefox <=3.6 - 5 only - // Old Firefox doesn't throw on a badly-escaped identifier. - el.querySelectorAll( "\\\f" ); - rbuggyQSA.push( "[\\r\\n\\f]" ); - } ); - - assert( function( el ) { - el.innerHTML = "" + - ""; - - // Support: Windows 8 Native Apps - // The type and name attributes are restricted during .innerHTML assignment - var input = document.createElement( "input" ); - input.setAttribute( "type", "hidden" ); - el.appendChild( input ).setAttribute( "name", "D" ); - - // Support: IE8 - // Enforce case-sensitivity of name attribute - if ( el.querySelectorAll( "[name=d]" ).length ) { - rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); - } - - // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) - // IE8 throws error here and will not see later tests - if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { - rbuggyQSA.push( ":enabled", ":disabled" ); - } - - // Support: IE9-11+ - // IE's :disabled selector does not pick up the children of disabled fieldsets - docElem.appendChild( el ).disabled = true; - if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { - rbuggyQSA.push( ":enabled", ":disabled" ); - } - - // Support: Opera 10 - 11 only - // Opera 10-11 does not throw on post-comma invalid pseudos - el.querySelectorAll( "*,:x" ); - rbuggyQSA.push( ",.*:" ); - } ); - } - - if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || - docElem.webkitMatchesSelector || - docElem.mozMatchesSelector || - docElem.oMatchesSelector || - docElem.msMatchesSelector ) ) ) ) { - - assert( function( el ) { - - // Check to see if it's possible to do matchesSelector - // on a disconnected node (IE 9) - support.disconnectedMatch = matches.call( el, "*" ); - - // This should fail with an exception - // Gecko does not error, returns false instead - matches.call( el, "[s!='']:x" ); - rbuggyMatches.push( "!=", pseudos ); - } ); - } - - rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); - rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); - - /* Contains - ---------------------------------------------------------------------- */ - hasCompare = rnative.test( docElem.compareDocumentPosition ); - - // Element contains another - // Purposefully self-exclusive - // As in, an element does not contain itself - contains = hasCompare || rnative.test( docElem.contains ) ? - function( a, b ) { - var adown = a.nodeType === 9 ? a.documentElement : a, - bup = b && b.parentNode; - return a === bup || !!( bup && bup.nodeType === 1 && ( - adown.contains ? - adown.contains( bup ) : - a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 - ) ); - } : - function( a, b ) { - if ( b ) { - while ( ( b = b.parentNode ) ) { - if ( b === a ) { - return true; - } - } - } - return false; - }; - - /* Sorting - ---------------------------------------------------------------------- */ - - // Document order sorting - sortOrder = hasCompare ? - function( a, b ) { - - // Flag for duplicate removal - if ( a === b ) { - hasDuplicate = true; - return 0; - } - - // Sort on method existence if only one input has compareDocumentPosition - var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; - if ( compare ) { - return compare; - } - - // Calculate position if both inputs belong to the same document - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? - a.compareDocumentPosition( b ) : - - // Otherwise we know they are disconnected - 1; - - // Disconnected nodes - if ( compare & 1 || - ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { - - // Choose the first element that is related to our preferred document - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( a == document || a.ownerDocument == preferredDoc && - contains( preferredDoc, a ) ) { - return -1; - } - - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( b == document || b.ownerDocument == preferredDoc && - contains( preferredDoc, b ) ) { - return 1; - } - - // Maintain original order - return sortInput ? - ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : - 0; - } - - return compare & 4 ? -1 : 1; - } : - function( a, b ) { - - // Exit early if the nodes are identical - if ( a === b ) { - hasDuplicate = true; - return 0; - } - - var cur, - i = 0, - aup = a.parentNode, - bup = b.parentNode, - ap = [ a ], - bp = [ b ]; - - // Parentless nodes are either documents or disconnected - if ( !aup || !bup ) { - - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - /* eslint-disable eqeqeq */ - return a == document ? -1 : - b == document ? 1 : - /* eslint-enable eqeqeq */ - aup ? -1 : - bup ? 1 : - sortInput ? - ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : - 0; - - // If the nodes are siblings, we can do a quick check - } else if ( aup === bup ) { - return siblingCheck( a, b ); - } - - // Otherwise we need full lists of their ancestors for comparison - cur = a; - while ( ( cur = cur.parentNode ) ) { - ap.unshift( cur ); - } - cur = b; - while ( ( cur = cur.parentNode ) ) { - bp.unshift( cur ); - } - - // Walk down the tree looking for a discrepancy - while ( ap[ i ] === bp[ i ] ) { - i++; - } - - return i ? - - // Do a sibling check if the nodes have a common ancestor - siblingCheck( ap[ i ], bp[ i ] ) : - - // Otherwise nodes in our document sort first - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - /* eslint-disable eqeqeq */ - ap[ i ] == preferredDoc ? -1 : - bp[ i ] == preferredDoc ? 1 : - /* eslint-enable eqeqeq */ - 0; - }; - - return document; -}; - -Sizzle.matches = function( expr, elements ) { - return Sizzle( expr, null, null, elements ); -}; - -Sizzle.matchesSelector = function( elem, expr ) { - setDocument( elem ); - - if ( support.matchesSelector && documentIsHTML && - !nonnativeSelectorCache[ expr + " " ] && - ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && - ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { - - try { - var ret = matches.call( elem, expr ); - - // IE 9's matchesSelector returns false on disconnected nodes - if ( ret || support.disconnectedMatch || - - // As well, disconnected nodes are said to be in a document - // fragment in IE 9 - elem.document && elem.document.nodeType !== 11 ) { - return ret; - } - } catch ( e ) { - nonnativeSelectorCache( expr, true ); - } - } - - return Sizzle( expr, document, null, [ elem ] ).length > 0; -}; - -Sizzle.contains = function( context, elem ) { - - // Set document vars if needed - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( ( context.ownerDocument || context ) != document ) { - setDocument( context ); - } - return contains( context, elem ); -}; - -Sizzle.attr = function( elem, name ) { - - // Set document vars if needed - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( ( elem.ownerDocument || elem ) != document ) { - setDocument( elem ); - } - - var fn = Expr.attrHandle[ name.toLowerCase() ], - - // Don't get fooled by Object.prototype properties (jQuery #13807) - val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? - fn( elem, name, !documentIsHTML ) : - undefined; - - return val !== undefined ? - val : - support.attributes || !documentIsHTML ? - elem.getAttribute( name ) : - ( val = elem.getAttributeNode( name ) ) && val.specified ? - val.value : - null; -}; - -Sizzle.escape = function( sel ) { - return ( sel + "" ).replace( rcssescape, fcssescape ); -}; - -Sizzle.error = function( msg ) { - throw new Error( "Syntax error, unrecognized expression: " + msg ); -}; - -/** - * Document sorting and removing duplicates - * @param {ArrayLike} results - */ -Sizzle.uniqueSort = function( results ) { - var elem, - duplicates = [], - j = 0, - i = 0; - - // Unless we *know* we can detect duplicates, assume their presence - hasDuplicate = !support.detectDuplicates; - sortInput = !support.sortStable && results.slice( 0 ); - results.sort( sortOrder ); - - if ( hasDuplicate ) { - while ( ( elem = results[ i++ ] ) ) { - if ( elem === results[ i ] ) { - j = duplicates.push( i ); - } - } - while ( j-- ) { - results.splice( duplicates[ j ], 1 ); - } - } - - // Clear input after sorting to release objects - // See https://github.com/jquery/sizzle/pull/225 - sortInput = null; - - return results; -}; - -/** - * Utility function for retrieving the text value of an array of DOM nodes - * @param {Array|Element} elem - */ -getText = Sizzle.getText = function( elem ) { - var node, - ret = "", - i = 0, - nodeType = elem.nodeType; - - if ( !nodeType ) { - - // If no nodeType, this is expected to be an array - while ( ( node = elem[ i++ ] ) ) { - - // Do not traverse comment nodes - ret += getText( node ); - } - } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { - - // Use textContent for elements - // innerText usage removed for consistency of new lines (jQuery #11153) - if ( typeof elem.textContent === "string" ) { - return elem.textContent; - } else { - - // Traverse its children - for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { - ret += getText( elem ); - } - } - } else if ( nodeType === 3 || nodeType === 4 ) { - return elem.nodeValue; - } - - // Do not include comment or processing instruction nodes - - return ret; -}; - -Expr = Sizzle.selectors = { - - // Can be adjusted by the user - cacheLength: 50, - - createPseudo: markFunction, - - match: matchExpr, - - attrHandle: {}, - - find: {}, - - relative: { - ">": { dir: "parentNode", first: true }, - " ": { dir: "parentNode" }, - "+": { dir: "previousSibling", first: true }, - "~": { dir: "previousSibling" } - }, - - preFilter: { - "ATTR": function( match ) { - match[ 1 ] = match[ 1 ].replace( runescape, funescape ); - - // Move the given value to match[3] whether quoted or unquoted - match[ 3 ] = ( match[ 3 ] || match[ 4 ] || - match[ 5 ] || "" ).replace( runescape, funescape ); - - if ( match[ 2 ] === "~=" ) { - match[ 3 ] = " " + match[ 3 ] + " "; - } - - return match.slice( 0, 4 ); - }, - - "CHILD": function( match ) { - - /* matches from matchExpr["CHILD"] - 1 type (only|nth|...) - 2 what (child|of-type) - 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) - 4 xn-component of xn+y argument ([+-]?\d*n|) - 5 sign of xn-component - 6 x of xn-component - 7 sign of y-component - 8 y of y-component - */ - match[ 1 ] = match[ 1 ].toLowerCase(); - - if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { - - // nth-* requires argument - if ( !match[ 3 ] ) { - Sizzle.error( match[ 0 ] ); - } - - // numeric x and y parameters for Expr.filter.CHILD - // remember that false/true cast respectively to 0/1 - match[ 4 ] = +( match[ 4 ] ? - match[ 5 ] + ( match[ 6 ] || 1 ) : - 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); - match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); - - // other types prohibit arguments - } else if ( match[ 3 ] ) { - Sizzle.error( match[ 0 ] ); - } - - return match; - }, - - "PSEUDO": function( match ) { - var excess, - unquoted = !match[ 6 ] && match[ 2 ]; - - if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { - return null; - } - - // Accept quoted arguments as-is - if ( match[ 3 ] ) { - match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; - - // Strip excess characters from unquoted arguments - } else if ( unquoted && rpseudo.test( unquoted ) && - - // Get excess from tokenize (recursively) - ( excess = tokenize( unquoted, true ) ) && - - // advance to the next closing parenthesis - ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { - - // excess is a negative index - match[ 0 ] = match[ 0 ].slice( 0, excess ); - match[ 2 ] = unquoted.slice( 0, excess ); - } - - // Return only captures needed by the pseudo filter method (type and argument) - return match.slice( 0, 3 ); - } - }, - - filter: { - - "TAG": function( nodeNameSelector ) { - var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); - return nodeNameSelector === "*" ? - function() { - return true; - } : - function( elem ) { - return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; - }; - }, - - "CLASS": function( className ) { - var pattern = classCache[ className + " " ]; - - return pattern || - ( pattern = new RegExp( "(^|" + whitespace + - ")" + className + "(" + whitespace + "|$)" ) ) && classCache( - className, function( elem ) { - return pattern.test( - typeof elem.className === "string" && elem.className || - typeof elem.getAttribute !== "undefined" && - elem.getAttribute( "class" ) || - "" - ); - } ); - }, - - "ATTR": function( name, operator, check ) { - return function( elem ) { - var result = Sizzle.attr( elem, name ); - - if ( result == null ) { - return operator === "!="; - } - if ( !operator ) { - return true; - } - - result += ""; - - /* eslint-disable max-len */ - - return operator === "=" ? result === check : - operator === "!=" ? result !== check : - operator === "^=" ? check && result.indexOf( check ) === 0 : - operator === "*=" ? check && result.indexOf( check ) > -1 : - operator === "$=" ? check && result.slice( -check.length ) === check : - operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : - operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : - false; - /* eslint-enable max-len */ - - }; - }, - - "CHILD": function( type, what, _argument, first, last ) { - var simple = type.slice( 0, 3 ) !== "nth", - forward = type.slice( -4 ) !== "last", - ofType = what === "of-type"; - - return first === 1 && last === 0 ? - - // Shortcut for :nth-*(n) - function( elem ) { - return !!elem.parentNode; - } : - - function( elem, _context, xml ) { - var cache, uniqueCache, outerCache, node, nodeIndex, start, - dir = simple !== forward ? "nextSibling" : "previousSibling", - parent = elem.parentNode, - name = ofType && elem.nodeName.toLowerCase(), - useCache = !xml && !ofType, - diff = false; - - if ( parent ) { - - // :(first|last|only)-(child|of-type) - if ( simple ) { - while ( dir ) { - node = elem; - while ( ( node = node[ dir ] ) ) { - if ( ofType ? - node.nodeName.toLowerCase() === name : - node.nodeType === 1 ) { - - return false; - } - } - - // Reverse direction for :only-* (if we haven't yet done so) - start = dir = type === "only" && !start && "nextSibling"; - } - return true; - } - - start = [ forward ? parent.firstChild : parent.lastChild ]; - - // non-xml :nth-child(...) stores cache data on `parent` - if ( forward && useCache ) { - - // Seek `elem` from a previously-cached index - - // ...in a gzip-friendly way - node = parent; - outerCache = node[ expando ] || ( node[ expando ] = {} ); - - // Support: IE <9 only - // Defend against cloned attroperties (jQuery gh-1709) - uniqueCache = outerCache[ node.uniqueID ] || - ( outerCache[ node.uniqueID ] = {} ); - - cache = uniqueCache[ type ] || []; - nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; - diff = nodeIndex && cache[ 2 ]; - node = nodeIndex && parent.childNodes[ nodeIndex ]; - - while ( ( node = ++nodeIndex && node && node[ dir ] || - - // Fallback to seeking `elem` from the start - ( diff = nodeIndex = 0 ) || start.pop() ) ) { - - // When found, cache indexes on `parent` and break - if ( node.nodeType === 1 && ++diff && node === elem ) { - uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; - break; - } - } - - } else { - - // Use previously-cached element index if available - if ( useCache ) { - - // ...in a gzip-friendly way - node = elem; - outerCache = node[ expando ] || ( node[ expando ] = {} ); - - // Support: IE <9 only - // Defend against cloned attroperties (jQuery gh-1709) - uniqueCache = outerCache[ node.uniqueID ] || - ( outerCache[ node.uniqueID ] = {} ); - - cache = uniqueCache[ type ] || []; - nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; - diff = nodeIndex; - } - - // xml :nth-child(...) - // or :nth-last-child(...) or :nth(-last)?-of-type(...) - if ( diff === false ) { - - // Use the same loop as above to seek `elem` from the start - while ( ( node = ++nodeIndex && node && node[ dir ] || - ( diff = nodeIndex = 0 ) || start.pop() ) ) { - - if ( ( ofType ? - node.nodeName.toLowerCase() === name : - node.nodeType === 1 ) && - ++diff ) { - - // Cache the index of each encountered element - if ( useCache ) { - outerCache = node[ expando ] || - ( node[ expando ] = {} ); - - // Support: IE <9 only - // Defend against cloned attroperties (jQuery gh-1709) - uniqueCache = outerCache[ node.uniqueID ] || - ( outerCache[ node.uniqueID ] = {} ); - - uniqueCache[ type ] = [ dirruns, diff ]; - } - - if ( node === elem ) { - break; - } - } - } - } - } - - // Incorporate the offset, then check against cycle size - diff -= last; - return diff === first || ( diff % first === 0 && diff / first >= 0 ); - } - }; - }, - - "PSEUDO": function( pseudo, argument ) { - - // pseudo-class names are case-insensitive - // http://www.w3.org/TR/selectors/#pseudo-classes - // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters - // Remember that setFilters inherits from pseudos - var args, - fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || - Sizzle.error( "unsupported pseudo: " + pseudo ); - - // The user may use createPseudo to indicate that - // arguments are needed to create the filter function - // just as Sizzle does - if ( fn[ expando ] ) { - return fn( argument ); - } - - // But maintain support for old signatures - if ( fn.length > 1 ) { - args = [ pseudo, pseudo, "", argument ]; - return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? - markFunction( function( seed, matches ) { - var idx, - matched = fn( seed, argument ), - i = matched.length; - while ( i-- ) { - idx = indexOf( seed, matched[ i ] ); - seed[ idx ] = !( matches[ idx ] = matched[ i ] ); - } - } ) : - function( elem ) { - return fn( elem, 0, args ); - }; - } - - return fn; - } - }, - - pseudos: { - - // Potentially complex pseudos - "not": markFunction( function( selector ) { - - // Trim the selector passed to compile - // to avoid treating leading and trailing - // spaces as combinators - var input = [], - results = [], - matcher = compile( selector.replace( rtrim, "$1" ) ); - - return matcher[ expando ] ? - markFunction( function( seed, matches, _context, xml ) { - var elem, - unmatched = matcher( seed, null, xml, [] ), - i = seed.length; - - // Match elements unmatched by `matcher` - while ( i-- ) { - if ( ( elem = unmatched[ i ] ) ) { - seed[ i ] = !( matches[ i ] = elem ); - } - } - } ) : - function( elem, _context, xml ) { - input[ 0 ] = elem; - matcher( input, null, xml, results ); - - // Don't keep the element (issue #299) - input[ 0 ] = null; - return !results.pop(); - }; - } ), - - "has": markFunction( function( selector ) { - return function( elem ) { - return Sizzle( selector, elem ).length > 0; - }; - } ), - - "contains": markFunction( function( text ) { - text = text.replace( runescape, funescape ); - return function( elem ) { - return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; - }; - } ), - - // "Whether an element is represented by a :lang() selector - // is based solely on the element's language value - // being equal to the identifier C, - // or beginning with the identifier C immediately followed by "-". - // The matching of C against the element's language value is performed case-insensitively. - // The identifier C does not have to be a valid language name." - // http://www.w3.org/TR/selectors/#lang-pseudo - "lang": markFunction( function( lang ) { - - // lang value must be a valid identifier - if ( !ridentifier.test( lang || "" ) ) { - Sizzle.error( "unsupported lang: " + lang ); - } - lang = lang.replace( runescape, funescape ).toLowerCase(); - return function( elem ) { - var elemLang; - do { - if ( ( elemLang = documentIsHTML ? - elem.lang : - elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { - - elemLang = elemLang.toLowerCase(); - return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; - } - } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); - return false; - }; - } ), - - // Miscellaneous - "target": function( elem ) { - var hash = window.location && window.location.hash; - return hash && hash.slice( 1 ) === elem.id; - }, - - "root": function( elem ) { - return elem === docElem; - }, - - "focus": function( elem ) { - return elem === document.activeElement && - ( !document.hasFocus || document.hasFocus() ) && - !!( elem.type || elem.href || ~elem.tabIndex ); - }, - - // Boolean properties - "enabled": createDisabledPseudo( false ), - "disabled": createDisabledPseudo( true ), - - "checked": function( elem ) { - - // In CSS3, :checked should return both checked and selected elements - // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked - var nodeName = elem.nodeName.toLowerCase(); - return ( nodeName === "input" && !!elem.checked ) || - ( nodeName === "option" && !!elem.selected ); - }, - - "selected": function( elem ) { - - // Accessing this property makes selected-by-default - // options in Safari work properly - if ( elem.parentNode ) { - // eslint-disable-next-line no-unused-expressions - elem.parentNode.selectedIndex; - } - - return elem.selected === true; - }, - - // Contents - "empty": function( elem ) { - - // http://www.w3.org/TR/selectors/#empty-pseudo - // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), - // but not by others (comment: 8; processing instruction: 7; etc.) - // nodeType < 6 works because attributes (2) do not appear as children - for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { - if ( elem.nodeType < 6 ) { - return false; - } - } - return true; - }, - - "parent": function( elem ) { - return !Expr.pseudos[ "empty" ]( elem ); - }, - - // Element/input types - "header": function( elem ) { - return rheader.test( elem.nodeName ); - }, - - "input": function( elem ) { - return rinputs.test( elem.nodeName ); - }, - - "button": function( elem ) { - var name = elem.nodeName.toLowerCase(); - return name === "input" && elem.type === "button" || name === "button"; - }, - - "text": function( elem ) { - var attr; - return elem.nodeName.toLowerCase() === "input" && - elem.type === "text" && - - // Support: IE<8 - // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" - ( ( attr = elem.getAttribute( "type" ) ) == null || - attr.toLowerCase() === "text" ); - }, - - // Position-in-collection - "first": createPositionalPseudo( function() { - return [ 0 ]; - } ), - - "last": createPositionalPseudo( function( _matchIndexes, length ) { - return [ length - 1 ]; - } ), - - "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { - return [ argument < 0 ? argument + length : argument ]; - } ), - - "even": createPositionalPseudo( function( matchIndexes, length ) { - var i = 0; - for ( ; i < length; i += 2 ) { - matchIndexes.push( i ); - } - return matchIndexes; - } ), - - "odd": createPositionalPseudo( function( matchIndexes, length ) { - var i = 1; - for ( ; i < length; i += 2 ) { - matchIndexes.push( i ); - } - return matchIndexes; - } ), - - "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { - var i = argument < 0 ? - argument + length : - argument > length ? - length : - argument; - for ( ; --i >= 0; ) { - matchIndexes.push( i ); - } - return matchIndexes; - } ), - - "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { - var i = argument < 0 ? argument + length : argument; - for ( ; ++i < length; ) { - matchIndexes.push( i ); - } - return matchIndexes; - } ) - } -}; - -Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; - -// Add button/input type pseudos -for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { - Expr.pseudos[ i ] = createInputPseudo( i ); -} -for ( i in { submit: true, reset: true } ) { - Expr.pseudos[ i ] = createButtonPseudo( i ); -} - -// Easy API for creating new setFilters -function setFilters() {} -setFilters.prototype = Expr.filters = Expr.pseudos; -Expr.setFilters = new setFilters(); - -tokenize = Sizzle.tokenize = function( selector, parseOnly ) { - var matched, match, tokens, type, - soFar, groups, preFilters, - cached = tokenCache[ selector + " " ]; - - if ( cached ) { - return parseOnly ? 0 : cached.slice( 0 ); - } - - soFar = selector; - groups = []; - preFilters = Expr.preFilter; - - while ( soFar ) { - - // Comma and first run - if ( !matched || ( match = rcomma.exec( soFar ) ) ) { - if ( match ) { - - // Don't consume trailing commas as valid - soFar = soFar.slice( match[ 0 ].length ) || soFar; - } - groups.push( ( tokens = [] ) ); - } - - matched = false; - - // Combinators - if ( ( match = rcombinators.exec( soFar ) ) ) { - matched = match.shift(); - tokens.push( { - value: matched, - - // Cast descendant combinators to space - type: match[ 0 ].replace( rtrim, " " ) - } ); - soFar = soFar.slice( matched.length ); - } - - // Filters - for ( type in Expr.filter ) { - if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || - ( match = preFilters[ type ]( match ) ) ) ) { - matched = match.shift(); - tokens.push( { - value: matched, - type: type, - matches: match - } ); - soFar = soFar.slice( matched.length ); - } - } - - if ( !matched ) { - break; - } - } - - // Return the length of the invalid excess - // if we're just parsing - // Otherwise, throw an error or return tokens - return parseOnly ? - soFar.length : - soFar ? - Sizzle.error( selector ) : - - // Cache the tokens - tokenCache( selector, groups ).slice( 0 ); -}; - -function toSelector( tokens ) { - var i = 0, - len = tokens.length, - selector = ""; - for ( ; i < len; i++ ) { - selector += tokens[ i ].value; - } - return selector; -} - -function addCombinator( matcher, combinator, base ) { - var dir = combinator.dir, - skip = combinator.next, - key = skip || dir, - checkNonElements = base && key === "parentNode", - doneName = done++; - - return combinator.first ? - - // Check against closest ancestor/preceding element - function( elem, context, xml ) { - while ( ( elem = elem[ dir ] ) ) { - if ( elem.nodeType === 1 || checkNonElements ) { - return matcher( elem, context, xml ); - } - } - return false; - } : - - // Check against all ancestor/preceding elements - function( elem, context, xml ) { - var oldCache, uniqueCache, outerCache, - newCache = [ dirruns, doneName ]; - - // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching - if ( xml ) { - while ( ( elem = elem[ dir ] ) ) { - if ( elem.nodeType === 1 || checkNonElements ) { - if ( matcher( elem, context, xml ) ) { - return true; - } - } - } - } else { - while ( ( elem = elem[ dir ] ) ) { - if ( elem.nodeType === 1 || checkNonElements ) { - outerCache = elem[ expando ] || ( elem[ expando ] = {} ); - - // Support: IE <9 only - // Defend against cloned attroperties (jQuery gh-1709) - uniqueCache = outerCache[ elem.uniqueID ] || - ( outerCache[ elem.uniqueID ] = {} ); - - if ( skip && skip === elem.nodeName.toLowerCase() ) { - elem = elem[ dir ] || elem; - } else if ( ( oldCache = uniqueCache[ key ] ) && - oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { - - // Assign to newCache so results back-propagate to previous elements - return ( newCache[ 2 ] = oldCache[ 2 ] ); - } else { - - // Reuse newcache so results back-propagate to previous elements - uniqueCache[ key ] = newCache; - - // A match means we're done; a fail means we have to keep checking - if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { - return true; - } - } - } - } - } - return false; - }; -} - -function elementMatcher( matchers ) { - return matchers.length > 1 ? - function( elem, context, xml ) { - var i = matchers.length; - while ( i-- ) { - if ( !matchers[ i ]( elem, context, xml ) ) { - return false; - } - } - return true; - } : - matchers[ 0 ]; -} - -function multipleContexts( selector, contexts, results ) { - var i = 0, - len = contexts.length; - for ( ; i < len; i++ ) { - Sizzle( selector, contexts[ i ], results ); - } - return results; -} - -function condense( unmatched, map, filter, context, xml ) { - var elem, - newUnmatched = [], - i = 0, - len = unmatched.length, - mapped = map != null; - - for ( ; i < len; i++ ) { - if ( ( elem = unmatched[ i ] ) ) { - if ( !filter || filter( elem, context, xml ) ) { - newUnmatched.push( elem ); - if ( mapped ) { - map.push( i ); - } - } - } - } - - return newUnmatched; -} - -function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { - if ( postFilter && !postFilter[ expando ] ) { - postFilter = setMatcher( postFilter ); - } - if ( postFinder && !postFinder[ expando ] ) { - postFinder = setMatcher( postFinder, postSelector ); - } - return markFunction( function( seed, results, context, xml ) { - var temp, i, elem, - preMap = [], - postMap = [], - preexisting = results.length, - - // Get initial elements from seed or context - elems = seed || multipleContexts( - selector || "*", - context.nodeType ? [ context ] : context, - [] - ), - - // Prefilter to get matcher input, preserving a map for seed-results synchronization - matcherIn = preFilter && ( seed || !selector ) ? - condense( elems, preMap, preFilter, context, xml ) : - elems, - - matcherOut = matcher ? - - // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, - postFinder || ( seed ? preFilter : preexisting || postFilter ) ? - - // ...intermediate processing is necessary - [] : - - // ...otherwise use results directly - results : - matcherIn; - - // Find primary matches - if ( matcher ) { - matcher( matcherIn, matcherOut, context, xml ); - } - - // Apply postFilter - if ( postFilter ) { - temp = condense( matcherOut, postMap ); - postFilter( temp, [], context, xml ); - - // Un-match failing elements by moving them back to matcherIn - i = temp.length; - while ( i-- ) { - if ( ( elem = temp[ i ] ) ) { - matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); - } - } - } - - if ( seed ) { - if ( postFinder || preFilter ) { - if ( postFinder ) { - - // Get the final matcherOut by condensing this intermediate into postFinder contexts - temp = []; - i = matcherOut.length; - while ( i-- ) { - if ( ( elem = matcherOut[ i ] ) ) { - - // Restore matcherIn since elem is not yet a final match - temp.push( ( matcherIn[ i ] = elem ) ); - } - } - postFinder( null, ( matcherOut = [] ), temp, xml ); - } - - // Move matched elements from seed to results to keep them synchronized - i = matcherOut.length; - while ( i-- ) { - if ( ( elem = matcherOut[ i ] ) && - ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { - - seed[ temp ] = !( results[ temp ] = elem ); - } - } - } - - // Add elements to results, through postFinder if defined - } else { - matcherOut = condense( - matcherOut === results ? - matcherOut.splice( preexisting, matcherOut.length ) : - matcherOut - ); - if ( postFinder ) { - postFinder( null, results, matcherOut, xml ); - } else { - push.apply( results, matcherOut ); - } - } - } ); -} - -function matcherFromTokens( tokens ) { - var checkContext, matcher, j, - len = tokens.length, - leadingRelative = Expr.relative[ tokens[ 0 ].type ], - implicitRelative = leadingRelative || Expr.relative[ " " ], - i = leadingRelative ? 1 : 0, - - // The foundational matcher ensures that elements are reachable from top-level context(s) - matchContext = addCombinator( function( elem ) { - return elem === checkContext; - }, implicitRelative, true ), - matchAnyContext = addCombinator( function( elem ) { - return indexOf( checkContext, elem ) > -1; - }, implicitRelative, true ), - matchers = [ function( elem, context, xml ) { - var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( - ( checkContext = context ).nodeType ? - matchContext( elem, context, xml ) : - matchAnyContext( elem, context, xml ) ); - - // Avoid hanging onto element (issue #299) - checkContext = null; - return ret; - } ]; - - for ( ; i < len; i++ ) { - if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { - matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; - } else { - matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); - - // Return special upon seeing a positional matcher - if ( matcher[ expando ] ) { - - // Find the next relative operator (if any) for proper handling - j = ++i; - for ( ; j < len; j++ ) { - if ( Expr.relative[ tokens[ j ].type ] ) { - break; - } - } - return setMatcher( - i > 1 && elementMatcher( matchers ), - i > 1 && toSelector( - - // If the preceding token was a descendant combinator, insert an implicit any-element `*` - tokens - .slice( 0, i - 1 ) - .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) - ).replace( rtrim, "$1" ), - matcher, - i < j && matcherFromTokens( tokens.slice( i, j ) ), - j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), - j < len && toSelector( tokens ) - ); - } - matchers.push( matcher ); - } - } - - return elementMatcher( matchers ); -} - -function matcherFromGroupMatchers( elementMatchers, setMatchers ) { - var bySet = setMatchers.length > 0, - byElement = elementMatchers.length > 0, - superMatcher = function( seed, context, xml, results, outermost ) { - var elem, j, matcher, - matchedCount = 0, - i = "0", - unmatched = seed && [], - setMatched = [], - contextBackup = outermostContext, - - // We must always have either seed elements or outermost context - elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), - - // Use integer dirruns iff this is the outermost matcher - dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), - len = elems.length; - - if ( outermost ) { - - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - outermostContext = context == document || context || outermost; - } - - // Add elements passing elementMatchers directly to results - // Support: IE<9, Safari - // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id - for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { - if ( byElement && elem ) { - j = 0; - - // Support: IE 11+, Edge 17 - 18+ - // IE/Edge sometimes throw a "Permission denied" error when strict-comparing - // two documents; shallow comparisons work. - // eslint-disable-next-line eqeqeq - if ( !context && elem.ownerDocument != document ) { - setDocument( elem ); - xml = !documentIsHTML; - } - while ( ( matcher = elementMatchers[ j++ ] ) ) { - if ( matcher( elem, context || document, xml ) ) { - results.push( elem ); - break; - } - } - if ( outermost ) { - dirruns = dirrunsUnique; - } - } - - // Track unmatched elements for set filters - if ( bySet ) { - - // They will have gone through all possible matchers - if ( ( elem = !matcher && elem ) ) { - matchedCount--; - } - - // Lengthen the array for every element, matched or not - if ( seed ) { - unmatched.push( elem ); - } - } - } - - // `i` is now the count of elements visited above, and adding it to `matchedCount` - // makes the latter nonnegative. - matchedCount += i; - - // Apply set filters to unmatched elements - // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` - // equals `i`), unless we didn't visit _any_ elements in the above loop because we have - // no element matchers and no seed. - // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that - // case, which will result in a "00" `matchedCount` that differs from `i` but is also - // numerically zero. - if ( bySet && i !== matchedCount ) { - j = 0; - while ( ( matcher = setMatchers[ j++ ] ) ) { - matcher( unmatched, setMatched, context, xml ); - } - - if ( seed ) { - - // Reintegrate element matches to eliminate the need for sorting - if ( matchedCount > 0 ) { - while ( i-- ) { - if ( !( unmatched[ i ] || setMatched[ i ] ) ) { - setMatched[ i ] = pop.call( results ); - } - } - } - - // Discard index placeholder values to get only actual matches - setMatched = condense( setMatched ); - } - - // Add matches to results - push.apply( results, setMatched ); - - // Seedless set matches succeeding multiple successful matchers stipulate sorting - if ( outermost && !seed && setMatched.length > 0 && - ( matchedCount + setMatchers.length ) > 1 ) { - - Sizzle.uniqueSort( results ); - } - } - - // Override manipulation of globals by nested matchers - if ( outermost ) { - dirruns = dirrunsUnique; - outermostContext = contextBackup; - } - - return unmatched; - }; - - return bySet ? - markFunction( superMatcher ) : - superMatcher; -} - -compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { - var i, - setMatchers = [], - elementMatchers = [], - cached = compilerCache[ selector + " " ]; - - if ( !cached ) { - - // Generate a function of recursive functions that can be used to check each element - if ( !match ) { - match = tokenize( selector ); - } - i = match.length; - while ( i-- ) { - cached = matcherFromTokens( match[ i ] ); - if ( cached[ expando ] ) { - setMatchers.push( cached ); - } else { - elementMatchers.push( cached ); - } - } - - // Cache the compiled function - cached = compilerCache( - selector, - matcherFromGroupMatchers( elementMatchers, setMatchers ) - ); - - // Save selector and tokenization - cached.selector = selector; - } - return cached; -}; - -/** - * A low-level selection function that works with Sizzle's compiled - * selector functions - * @param {String|Function} selector A selector or a pre-compiled - * selector function built with Sizzle.compile - * @param {Element} context - * @param {Array} [results] - * @param {Array} [seed] A set of elements to match against - */ -select = Sizzle.select = function( selector, context, results, seed ) { - var i, tokens, token, type, find, - compiled = typeof selector === "function" && selector, - match = !seed && tokenize( ( selector = compiled.selector || selector ) ); - - results = results || []; - - // Try to minimize operations if there is only one selector in the list and no seed - // (the latter of which guarantees us context) - if ( match.length === 1 ) { - - // Reduce context if the leading compound selector is an ID - tokens = match[ 0 ] = match[ 0 ].slice( 0 ); - if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && - context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { - - context = ( Expr.find[ "ID" ]( token.matches[ 0 ] - .replace( runescape, funescape ), context ) || [] )[ 0 ]; - if ( !context ) { - return results; - - // Precompiled matchers will still verify ancestry, so step up a level - } else if ( compiled ) { - context = context.parentNode; - } - - selector = selector.slice( tokens.shift().value.length ); - } - - // Fetch a seed set for right-to-left matching - i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; - while ( i-- ) { - token = tokens[ i ]; - - // Abort if we hit a combinator - if ( Expr.relative[ ( type = token.type ) ] ) { - break; - } - if ( ( find = Expr.find[ type ] ) ) { - - // Search, expanding context for leading sibling combinators - if ( ( seed = find( - token.matches[ 0 ].replace( runescape, funescape ), - rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || - context - ) ) ) { - - // If seed is empty or no tokens remain, we can return early - tokens.splice( i, 1 ); - selector = seed.length && toSelector( tokens ); - if ( !selector ) { - push.apply( results, seed ); - return results; - } - - break; - } - } - } - } - - // Compile and execute a filtering function if one is not provided - // Provide `match` to avoid retokenization if we modified the selector above - ( compiled || compile( selector, match ) )( - seed, - context, - !documentIsHTML, - results, - !context || rsibling.test( selector ) && testContext( context.parentNode ) || context - ); - return results; -}; - -// One-time assignments - -// Sort stability -support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; - -// Support: Chrome 14-35+ -// Always assume duplicates if they aren't passed to the comparison function -support.detectDuplicates = !!hasDuplicate; - -// Initialize against the default document -setDocument(); - -// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) -// Detached nodes confoundingly follow *each other* -support.sortDetached = assert( function( el ) { - - // Should return 1, but returns 4 (following) - return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; -} ); - -// Support: IE<8 -// Prevent attribute/property "interpolation" -// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx -if ( !assert( function( el ) { - el.innerHTML = ""; - return el.firstChild.getAttribute( "href" ) === "#"; -} ) ) { - addHandle( "type|href|height|width", function( elem, name, isXML ) { - if ( !isXML ) { - return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); - } - } ); -} - -// Support: IE<9 -// Use defaultValue in place of getAttribute("value") -if ( !support.attributes || !assert( function( el ) { - el.innerHTML = ""; - el.firstChild.setAttribute( "value", "" ); - return el.firstChild.getAttribute( "value" ) === ""; -} ) ) { - addHandle( "value", function( elem, _name, isXML ) { - if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { - return elem.defaultValue; - } - } ); -} - -// Support: IE<9 -// Use getAttributeNode to fetch booleans when getAttribute lies -if ( !assert( function( el ) { - return el.getAttribute( "disabled" ) == null; -} ) ) { - addHandle( booleans, function( elem, name, isXML ) { - var val; - if ( !isXML ) { - return elem[ name ] === true ? name.toLowerCase() : - ( val = elem.getAttributeNode( name ) ) && val.specified ? - val.value : - null; - } - } ); -} - -return Sizzle; - -} )( window ); - - - -jQuery.find = Sizzle; -jQuery.expr = Sizzle.selectors; - -// Deprecated -jQuery.expr[ ":" ] = jQuery.expr.pseudos; -jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; -jQuery.text = Sizzle.getText; -jQuery.isXMLDoc = Sizzle.isXML; -jQuery.contains = Sizzle.contains; -jQuery.escapeSelector = Sizzle.escape; - - - - -var dir = function( elem, dir, until ) { - var matched = [], - truncate = until !== undefined; - - while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { - if ( elem.nodeType === 1 ) { - if ( truncate && jQuery( elem ).is( until ) ) { - break; - } - matched.push( elem ); - } - } - return matched; -}; - - -var siblings = function( n, elem ) { - var matched = []; - - for ( ; n; n = n.nextSibling ) { - if ( n.nodeType === 1 && n !== elem ) { - matched.push( n ); - } - } - - return matched; -}; - - -var rneedsContext = jQuery.expr.match.needsContext; - - - -function nodeName( elem, name ) { - - return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); - -} -var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); - - - -// Implement the identical functionality for filter and not -function winnow( elements, qualifier, not ) { - if ( isFunction( qualifier ) ) { - return jQuery.grep( elements, function( elem, i ) { - return !!qualifier.call( elem, i, elem ) !== not; - } ); - } - - // Single element - if ( qualifier.nodeType ) { - return jQuery.grep( elements, function( elem ) { - return ( elem === qualifier ) !== not; - } ); - } - - // Arraylike of elements (jQuery, arguments, Array) - if ( typeof qualifier !== "string" ) { - return jQuery.grep( elements, function( elem ) { - return ( indexOf.call( qualifier, elem ) > -1 ) !== not; - } ); - } - - // Filtered directly for both simple and complex selectors - return jQuery.filter( qualifier, elements, not ); -} - -jQuery.filter = function( expr, elems, not ) { - var elem = elems[ 0 ]; - - if ( not ) { - expr = ":not(" + expr + ")"; - } - - if ( elems.length === 1 && elem.nodeType === 1 ) { - return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; - } - - return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { - return elem.nodeType === 1; - } ) ); -}; - -jQuery.fn.extend( { - find: function( selector ) { - var i, ret, - len = this.length, - self = this; - - if ( typeof selector !== "string" ) { - return this.pushStack( jQuery( selector ).filter( function() { - for ( i = 0; i < len; i++ ) { - if ( jQuery.contains( self[ i ], this ) ) { - return true; - } - } - } ) ); - } - - ret = this.pushStack( [] ); - - for ( i = 0; i < len; i++ ) { - jQuery.find( selector, self[ i ], ret ); - } - - return len > 1 ? jQuery.uniqueSort( ret ) : ret; - }, - filter: function( selector ) { - return this.pushStack( winnow( this, selector || [], false ) ); - }, - not: function( selector ) { - return this.pushStack( winnow( this, selector || [], true ) ); - }, - is: function( selector ) { - return !!winnow( - this, - - // If this is a positional/relative selector, check membership in the returned set - // so $("p:first").is("p:last") won't return true for a doc with two "p". - typeof selector === "string" && rneedsContext.test( selector ) ? - jQuery( selector ) : - selector || [], - false - ).length; - } -} ); - - -// Initialize a jQuery object - - -// A central reference to the root jQuery(document) -var rootjQuery, - - // A simple way to check for HTML strings - // Prioritize #id over to avoid XSS via location.hash (#9521) - // Strict HTML recognition (#11290: must start with <) - // Shortcut simple #id case for speed - rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, - - init = jQuery.fn.init = function( selector, context, root ) { - var match, elem; - - // HANDLE: $(""), $(null), $(undefined), $(false) - if ( !selector ) { - return this; - } - - // Method init() accepts an alternate rootjQuery - // so migrate can support jQuery.sub (gh-2101) - root = root || rootjQuery; - - // Handle HTML strings - if ( typeof selector === "string" ) { - if ( selector[ 0 ] === "<" && - selector[ selector.length - 1 ] === ">" && - selector.length >= 3 ) { - - // Assume that strings that start and end with <> are HTML and skip the regex check - match = [ null, selector, null ]; - - } else { - match = rquickExpr.exec( selector ); - } - - // Match html or make sure no context is specified for #id - if ( match && ( match[ 1 ] || !context ) ) { - - // HANDLE: $(html) -> $(array) - if ( match[ 1 ] ) { - context = context instanceof jQuery ? context[ 0 ] : context; - - // Option to run scripts is true for back-compat - // Intentionally let the error be thrown if parseHTML is not present - jQuery.merge( this, jQuery.parseHTML( - match[ 1 ], - context && context.nodeType ? context.ownerDocument || context : document, - true - ) ); - - // HANDLE: $(html, props) - if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { - for ( match in context ) { - - // Properties of context are called as methods if possible - if ( isFunction( this[ match ] ) ) { - this[ match ]( context[ match ] ); - - // ...and otherwise set as attributes - } else { - this.attr( match, context[ match ] ); - } - } - } - - return this; - - // HANDLE: $(#id) - } else { - elem = document.getElementById( match[ 2 ] ); - - if ( elem ) { - - // Inject the element directly into the jQuery object - this[ 0 ] = elem; - this.length = 1; - } - return this; - } - - // HANDLE: $(expr, $(...)) - } else if ( !context || context.jquery ) { - return ( context || root ).find( selector ); - - // HANDLE: $(expr, context) - // (which is just equivalent to: $(context).find(expr) - } else { - return this.constructor( context ).find( selector ); - } - - // HANDLE: $(DOMElement) - } else if ( selector.nodeType ) { - this[ 0 ] = selector; - this.length = 1; - return this; - - // HANDLE: $(function) - // Shortcut for document ready - } else if ( isFunction( selector ) ) { - return root.ready !== undefined ? - root.ready( selector ) : - - // Execute immediately if ready is not present - selector( jQuery ); - } - - return jQuery.makeArray( selector, this ); - }; - -// Give the init function the jQuery prototype for later instantiation -init.prototype = jQuery.fn; - -// Initialize central reference -rootjQuery = jQuery( document ); - - -var rparentsprev = /^(?:parents|prev(?:Until|All))/, - - // Methods guaranteed to produce a unique set when starting from a unique set - guaranteedUnique = { - children: true, - contents: true, - next: true, - prev: true - }; - -jQuery.fn.extend( { - has: function( target ) { - var targets = jQuery( target, this ), - l = targets.length; - - return this.filter( function() { - var i = 0; - for ( ; i < l; i++ ) { - if ( jQuery.contains( this, targets[ i ] ) ) { - return true; - } - } - } ); - }, - - closest: function( selectors, context ) { - var cur, - i = 0, - l = this.length, - matched = [], - targets = typeof selectors !== "string" && jQuery( selectors ); - - // Positional selectors never match, since there's no _selection_ context - if ( !rneedsContext.test( selectors ) ) { - for ( ; i < l; i++ ) { - for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { - - // Always skip document fragments - if ( cur.nodeType < 11 && ( targets ? - targets.index( cur ) > -1 : - - // Don't pass non-elements to Sizzle - cur.nodeType === 1 && - jQuery.find.matchesSelector( cur, selectors ) ) ) { - - matched.push( cur ); - break; - } - } - } - } - - return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); - }, - - // Determine the position of an element within the set - index: function( elem ) { - - // No argument, return index in parent - if ( !elem ) { - return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; - } - - // Index in selector - if ( typeof elem === "string" ) { - return indexOf.call( jQuery( elem ), this[ 0 ] ); - } - - // Locate the position of the desired element - return indexOf.call( this, - - // If it receives a jQuery object, the first element is used - elem.jquery ? elem[ 0 ] : elem - ); - }, - - add: function( selector, context ) { - return this.pushStack( - jQuery.uniqueSort( - jQuery.merge( this.get(), jQuery( selector, context ) ) - ) - ); - }, - - addBack: function( selector ) { - return this.add( selector == null ? - this.prevObject : this.prevObject.filter( selector ) - ); - } -} ); - -function sibling( cur, dir ) { - while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} - return cur; -} - -jQuery.each( { - parent: function( elem ) { - var parent = elem.parentNode; - return parent && parent.nodeType !== 11 ? parent : null; - }, - parents: function( elem ) { - return dir( elem, "parentNode" ); - }, - parentsUntil: function( elem, _i, until ) { - return dir( elem, "parentNode", until ); - }, - next: function( elem ) { - return sibling( elem, "nextSibling" ); - }, - prev: function( elem ) { - return sibling( elem, "previousSibling" ); - }, - nextAll: function( elem ) { - return dir( elem, "nextSibling" ); - }, - prevAll: function( elem ) { - return dir( elem, "previousSibling" ); - }, - nextUntil: function( elem, _i, until ) { - return dir( elem, "nextSibling", until ); - }, - prevUntil: function( elem, _i, until ) { - return dir( elem, "previousSibling", until ); - }, - siblings: function( elem ) { - return siblings( ( elem.parentNode || {} ).firstChild, elem ); - }, - children: function( elem ) { - return siblings( elem.firstChild ); - }, - contents: function( elem ) { - if ( elem.contentDocument != null && - - // Support: IE 11+ - // elements with no `data` attribute has an object - // `contentDocument` with a `null` prototype. - getProto( elem.contentDocument ) ) { - - return elem.contentDocument; - } - - // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only - // Treat the template element as a regular one in browsers that - // don't support it. - if ( nodeName( elem, "template" ) ) { - elem = elem.content || elem; - } - - return jQuery.merge( [], elem.childNodes ); - } -}, function( name, fn ) { - jQuery.fn[ name ] = function( until, selector ) { - var matched = jQuery.map( this, fn, until ); - - if ( name.slice( -5 ) !== "Until" ) { - selector = until; - } - - if ( selector && typeof selector === "string" ) { - matched = jQuery.filter( selector, matched ); - } - - if ( this.length > 1 ) { - - // Remove duplicates - if ( !guaranteedUnique[ name ] ) { - jQuery.uniqueSort( matched ); - } - - // Reverse order for parents* and prev-derivatives - if ( rparentsprev.test( name ) ) { - matched.reverse(); - } - } - - return this.pushStack( matched ); - }; -} ); -var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); - - - -// Convert String-formatted options into Object-formatted ones -function createOptions( options ) { - var object = {}; - jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { - object[ flag ] = true; - } ); - return object; -} - -/* - * Create a callback list using the following parameters: - * - * options: an optional list of space-separated options that will change how - * the callback list behaves or a more traditional option object - * - * By default a callback list will act like an event callback list and can be - * "fired" multiple times. - * - * Possible options: - * - * once: will ensure the callback list can only be fired once (like a Deferred) - * - * memory: will keep track of previous values and will call any callback added - * after the list has been fired right away with the latest "memorized" - * values (like a Deferred) - * - * unique: will ensure a callback can only be added once (no duplicate in the list) - * - * stopOnFalse: interrupt callings when a callback returns false - * - */ -jQuery.Callbacks = function( options ) { - - // Convert options from String-formatted to Object-formatted if needed - // (we check in cache first) - options = typeof options === "string" ? - createOptions( options ) : - jQuery.extend( {}, options ); - - var // Flag to know if list is currently firing - firing, - - // Last fire value for non-forgettable lists - memory, - - // Flag to know if list was already fired - fired, - - // Flag to prevent firing - locked, - - // Actual callback list - list = [], - - // Queue of execution data for repeatable lists - queue = [], - - // Index of currently firing callback (modified by add/remove as needed) - firingIndex = -1, - - // Fire callbacks - fire = function() { - - // Enforce single-firing - locked = locked || options.once; - - // Execute callbacks for all pending executions, - // respecting firingIndex overrides and runtime changes - fired = firing = true; - for ( ; queue.length; firingIndex = -1 ) { - memory = queue.shift(); - while ( ++firingIndex < list.length ) { - - // Run callback and check for early termination - if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && - options.stopOnFalse ) { - - // Jump to end and forget the data so .add doesn't re-fire - firingIndex = list.length; - memory = false; - } - } - } - - // Forget the data if we're done with it - if ( !options.memory ) { - memory = false; - } - - firing = false; - - // Clean up if we're done firing for good - if ( locked ) { - - // Keep an empty list if we have data for future add calls - if ( memory ) { - list = []; - - // Otherwise, this object is spent - } else { - list = ""; - } - } - }, - - // Actual Callbacks object - self = { - - // Add a callback or a collection of callbacks to the list - add: function() { - if ( list ) { - - // If we have memory from a past run, we should fire after adding - if ( memory && !firing ) { - firingIndex = list.length - 1; - queue.push( memory ); - } - - ( function add( args ) { - jQuery.each( args, function( _, arg ) { - if ( isFunction( arg ) ) { - if ( !options.unique || !self.has( arg ) ) { - list.push( arg ); - } - } else if ( arg && arg.length && toType( arg ) !== "string" ) { - - // Inspect recursively - add( arg ); - } - } ); - } )( arguments ); - - if ( memory && !firing ) { - fire(); - } - } - return this; - }, - - // Remove a callback from the list - remove: function() { - jQuery.each( arguments, function( _, arg ) { - var index; - while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { - list.splice( index, 1 ); - - // Handle firing indexes - if ( index <= firingIndex ) { - firingIndex--; - } - } - } ); - return this; - }, - - // Check if a given callback is in the list. - // If no argument is given, return whether or not list has callbacks attached. - has: function( fn ) { - return fn ? - jQuery.inArray( fn, list ) > -1 : - list.length > 0; - }, - - // Remove all callbacks from the list - empty: function() { - if ( list ) { - list = []; - } - return this; - }, - - // Disable .fire and .add - // Abort any current/pending executions - // Clear all callbacks and values - disable: function() { - locked = queue = []; - list = memory = ""; - return this; - }, - disabled: function() { - return !list; - }, - - // Disable .fire - // Also disable .add unless we have memory (since it would have no effect) - // Abort any pending executions - lock: function() { - locked = queue = []; - if ( !memory && !firing ) { - list = memory = ""; - } - return this; - }, - locked: function() { - return !!locked; - }, - - // Call all callbacks with the given context and arguments - fireWith: function( context, args ) { - if ( !locked ) { - args = args || []; - args = [ context, args.slice ? args.slice() : args ]; - queue.push( args ); - if ( !firing ) { - fire(); - } - } - return this; - }, - - // Call all the callbacks with the given arguments - fire: function() { - self.fireWith( this, arguments ); - return this; - }, - - // To know if the callbacks have already been called at least once - fired: function() { - return !!fired; - } - }; - - return self; -}; - - -function Identity( v ) { - return v; -} -function Thrower( ex ) { - throw ex; -} - -function adoptValue( value, resolve, reject, noValue ) { - var method; - - try { - - // Check for promise aspect first to privilege synchronous behavior - if ( value && isFunction( ( method = value.promise ) ) ) { - method.call( value ).done( resolve ).fail( reject ); - - // Other thenables - } else if ( value && isFunction( ( method = value.then ) ) ) { - method.call( value, resolve, reject ); - - // Other non-thenables - } else { - - // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: - // * false: [ value ].slice( 0 ) => resolve( value ) - // * true: [ value ].slice( 1 ) => resolve() - resolve.apply( undefined, [ value ].slice( noValue ) ); - } - - // For Promises/A+, convert exceptions into rejections - // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in - // Deferred#then to conditionally suppress rejection. - } catch ( value ) { - - // Support: Android 4.0 only - // Strict mode functions invoked without .call/.apply get global-object context - reject.apply( undefined, [ value ] ); - } -} - -jQuery.extend( { - - Deferred: function( func ) { - var tuples = [ - - // action, add listener, callbacks, - // ... .then handlers, argument index, [final state] - [ "notify", "progress", jQuery.Callbacks( "memory" ), - jQuery.Callbacks( "memory" ), 2 ], - [ "resolve", "done", jQuery.Callbacks( "once memory" ), - jQuery.Callbacks( "once memory" ), 0, "resolved" ], - [ "reject", "fail", jQuery.Callbacks( "once memory" ), - jQuery.Callbacks( "once memory" ), 1, "rejected" ] - ], - state = "pending", - promise = { - state: function() { - return state; - }, - always: function() { - deferred.done( arguments ).fail( arguments ); - return this; - }, - "catch": function( fn ) { - return promise.then( null, fn ); - }, - - // Keep pipe for back-compat - pipe: function( /* fnDone, fnFail, fnProgress */ ) { - var fns = arguments; - - return jQuery.Deferred( function( newDefer ) { - jQuery.each( tuples, function( _i, tuple ) { - - // Map tuples (progress, done, fail) to arguments (done, fail, progress) - var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; - - // deferred.progress(function() { bind to newDefer or newDefer.notify }) - // deferred.done(function() { bind to newDefer or newDefer.resolve }) - // deferred.fail(function() { bind to newDefer or newDefer.reject }) - deferred[ tuple[ 1 ] ]( function() { - var returned = fn && fn.apply( this, arguments ); - if ( returned && isFunction( returned.promise ) ) { - returned.promise() - .progress( newDefer.notify ) - .done( newDefer.resolve ) - .fail( newDefer.reject ); - } else { - newDefer[ tuple[ 0 ] + "With" ]( - this, - fn ? [ returned ] : arguments - ); - } - } ); - } ); - fns = null; - } ).promise(); - }, - then: function( onFulfilled, onRejected, onProgress ) { - var maxDepth = 0; - function resolve( depth, deferred, handler, special ) { - return function() { - var that = this, - args = arguments, - mightThrow = function() { - var returned, then; - - // Support: Promises/A+ section 2.3.3.3.3 - // https://promisesaplus.com/#point-59 - // Ignore double-resolution attempts - if ( depth < maxDepth ) { - return; - } - - returned = handler.apply( that, args ); - - // Support: Promises/A+ section 2.3.1 - // https://promisesaplus.com/#point-48 - if ( returned === deferred.promise() ) { - throw new TypeError( "Thenable self-resolution" ); - } - - // Support: Promises/A+ sections 2.3.3.1, 3.5 - // https://promisesaplus.com/#point-54 - // https://promisesaplus.com/#point-75 - // Retrieve `then` only once - then = returned && - - // Support: Promises/A+ section 2.3.4 - // https://promisesaplus.com/#point-64 - // Only check objects and functions for thenability - ( typeof returned === "object" || - typeof returned === "function" ) && - returned.then; - - // Handle a returned thenable - if ( isFunction( then ) ) { - - // Special processors (notify) just wait for resolution - if ( special ) { - then.call( - returned, - resolve( maxDepth, deferred, Identity, special ), - resolve( maxDepth, deferred, Thrower, special ) - ); - - // Normal processors (resolve) also hook into progress - } else { - - // ...and disregard older resolution values - maxDepth++; - - then.call( - returned, - resolve( maxDepth, deferred, Identity, special ), - resolve( maxDepth, deferred, Thrower, special ), - resolve( maxDepth, deferred, Identity, - deferred.notifyWith ) - ); - } - - // Handle all other returned values - } else { - - // Only substitute handlers pass on context - // and multiple values (non-spec behavior) - if ( handler !== Identity ) { - that = undefined; - args = [ returned ]; - } - - // Process the value(s) - // Default process is resolve - ( special || deferred.resolveWith )( that, args ); - } - }, - - // Only normal processors (resolve) catch and reject exceptions - process = special ? - mightThrow : - function() { - try { - mightThrow(); - } catch ( e ) { - - if ( jQuery.Deferred.exceptionHook ) { - jQuery.Deferred.exceptionHook( e, - process.stackTrace ); - } - - // Support: Promises/A+ section 2.3.3.3.4.1 - // https://promisesaplus.com/#point-61 - // Ignore post-resolution exceptions - if ( depth + 1 >= maxDepth ) { - - // Only substitute handlers pass on context - // and multiple values (non-spec behavior) - if ( handler !== Thrower ) { - that = undefined; - args = [ e ]; - } - - deferred.rejectWith( that, args ); - } - } - }; - - // Support: Promises/A+ section 2.3.3.3.1 - // https://promisesaplus.com/#point-57 - // Re-resolve promises immediately to dodge false rejection from - // subsequent errors - if ( depth ) { - process(); - } else { - - // Call an optional hook to record the stack, in case of exception - // since it's otherwise lost when execution goes async - if ( jQuery.Deferred.getStackHook ) { - process.stackTrace = jQuery.Deferred.getStackHook(); - } - window.setTimeout( process ); - } - }; - } - - return jQuery.Deferred( function( newDefer ) { - - // progress_handlers.add( ... ) - tuples[ 0 ][ 3 ].add( - resolve( - 0, - newDefer, - isFunction( onProgress ) ? - onProgress : - Identity, - newDefer.notifyWith - ) - ); - - // fulfilled_handlers.add( ... ) - tuples[ 1 ][ 3 ].add( - resolve( - 0, - newDefer, - isFunction( onFulfilled ) ? - onFulfilled : - Identity - ) - ); - - // rejected_handlers.add( ... ) - tuples[ 2 ][ 3 ].add( - resolve( - 0, - newDefer, - isFunction( onRejected ) ? - onRejected : - Thrower - ) - ); - } ).promise(); - }, - - // Get a promise for this deferred - // If obj is provided, the promise aspect is added to the object - promise: function( obj ) { - return obj != null ? jQuery.extend( obj, promise ) : promise; - } - }, - deferred = {}; - - // Add list-specific methods - jQuery.each( tuples, function( i, tuple ) { - var list = tuple[ 2 ], - stateString = tuple[ 5 ]; - - // promise.progress = list.add - // promise.done = list.add - // promise.fail = list.add - promise[ tuple[ 1 ] ] = list.add; - - // Handle state - if ( stateString ) { - list.add( - function() { - - // state = "resolved" (i.e., fulfilled) - // state = "rejected" - state = stateString; - }, - - // rejected_callbacks.disable - // fulfilled_callbacks.disable - tuples[ 3 - i ][ 2 ].disable, - - // rejected_handlers.disable - // fulfilled_handlers.disable - tuples[ 3 - i ][ 3 ].disable, - - // progress_callbacks.lock - tuples[ 0 ][ 2 ].lock, - - // progress_handlers.lock - tuples[ 0 ][ 3 ].lock - ); - } - - // progress_handlers.fire - // fulfilled_handlers.fire - // rejected_handlers.fire - list.add( tuple[ 3 ].fire ); - - // deferred.notify = function() { deferred.notifyWith(...) } - // deferred.resolve = function() { deferred.resolveWith(...) } - // deferred.reject = function() { deferred.rejectWith(...) } - deferred[ tuple[ 0 ] ] = function() { - deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); - return this; - }; - - // deferred.notifyWith = list.fireWith - // deferred.resolveWith = list.fireWith - // deferred.rejectWith = list.fireWith - deferred[ tuple[ 0 ] + "With" ] = list.fireWith; - } ); - - // Make the deferred a promise - promise.promise( deferred ); - - // Call given func if any - if ( func ) { - func.call( deferred, deferred ); - } - - // All done! - return deferred; - }, - - // Deferred helper - when: function( singleValue ) { - var - - // count of uncompleted subordinates - remaining = arguments.length, - - // count of unprocessed arguments - i = remaining, - - // subordinate fulfillment data - resolveContexts = Array( i ), - resolveValues = slice.call( arguments ), - - // the primary Deferred - primary = jQuery.Deferred(), - - // subordinate callback factory - updateFunc = function( i ) { - return function( value ) { - resolveContexts[ i ] = this; - resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; - if ( !( --remaining ) ) { - primary.resolveWith( resolveContexts, resolveValues ); - } - }; - }; - - // Single- and empty arguments are adopted like Promise.resolve - if ( remaining <= 1 ) { - adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, - !remaining ); - - // Use .then() to unwrap secondary thenables (cf. gh-3000) - if ( primary.state() === "pending" || - isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { - - return primary.then(); - } - } - - // Multiple arguments are aggregated like Promise.all array elements - while ( i-- ) { - adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); - } - - return primary.promise(); - } -} ); - - -// These usually indicate a programmer mistake during development, -// warn about them ASAP rather than swallowing them by default. -var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; - -jQuery.Deferred.exceptionHook = function( error, stack ) { - - // Support: IE 8 - 9 only - // Console exists when dev tools are open, which can happen at any time - if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { - window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); - } -}; - - - - -jQuery.readyException = function( error ) { - window.setTimeout( function() { - throw error; - } ); -}; - - - - -// The deferred used on DOM ready -var readyList = jQuery.Deferred(); - -jQuery.fn.ready = function( fn ) { - - readyList - .then( fn ) - - // Wrap jQuery.readyException in a function so that the lookup - // happens at the time of error handling instead of callback - // registration. - .catch( function( error ) { - jQuery.readyException( error ); - } ); - - return this; -}; - -jQuery.extend( { - - // Is the DOM ready to be used? Set to true once it occurs. - isReady: false, - - // A counter to track how many items to wait for before - // the ready event fires. See #6781 - readyWait: 1, - - // Handle when the DOM is ready - ready: function( wait ) { - - // Abort if there are pending holds or we're already ready - if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { - return; - } - - // Remember that the DOM is ready - jQuery.isReady = true; - - // If a normal DOM Ready event fired, decrement, and wait if need be - if ( wait !== true && --jQuery.readyWait > 0 ) { - return; - } - - // If there are functions bound, to execute - readyList.resolveWith( document, [ jQuery ] ); - } -} ); - -jQuery.ready.then = readyList.then; - -// The ready event handler and self cleanup method -function completed() { - document.removeEventListener( "DOMContentLoaded", completed ); - window.removeEventListener( "load", completed ); - jQuery.ready(); -} - -// Catch cases where $(document).ready() is called -// after the browser event has already occurred. -// Support: IE <=9 - 10 only -// Older IE sometimes signals "interactive" too soon -if ( document.readyState === "complete" || - ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { - - // Handle it asynchronously to allow scripts the opportunity to delay ready - window.setTimeout( jQuery.ready ); - -} else { - - // Use the handy event callback - document.addEventListener( "DOMContentLoaded", completed ); - - // A fallback to window.onload, that will always work - window.addEventListener( "load", completed ); -} - - - - -// Multifunctional method to get and set values of a collection -// The value/s can optionally be executed if it's a function -var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { - var i = 0, - len = elems.length, - bulk = key == null; - - // Sets many values - if ( toType( key ) === "object" ) { - chainable = true; - for ( i in key ) { - access( elems, fn, i, key[ i ], true, emptyGet, raw ); - } - - // Sets one value - } else if ( value !== undefined ) { - chainable = true; - - if ( !isFunction( value ) ) { - raw = true; - } - - if ( bulk ) { - - // Bulk operations run against the entire set - if ( raw ) { - fn.call( elems, value ); - fn = null; - - // ...except when executing function values - } else { - bulk = fn; - fn = function( elem, _key, value ) { - return bulk.call( jQuery( elem ), value ); - }; - } - } - - if ( fn ) { - for ( ; i < len; i++ ) { - fn( - elems[ i ], key, raw ? - value : - value.call( elems[ i ], i, fn( elems[ i ], key ) ) - ); - } - } - } - - if ( chainable ) { - return elems; - } - - // Gets - if ( bulk ) { - return fn.call( elems ); - } - - return len ? fn( elems[ 0 ], key ) : emptyGet; -}; - - -// Matches dashed string for camelizing -var rmsPrefix = /^-ms-/, - rdashAlpha = /-([a-z])/g; - -// Used by camelCase as callback to replace() -function fcamelCase( _all, letter ) { - return letter.toUpperCase(); -} - -// Convert dashed to camelCase; used by the css and data modules -// Support: IE <=9 - 11, Edge 12 - 15 -// Microsoft forgot to hump their vendor prefix (#9572) -function camelCase( string ) { - return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); -} -var acceptData = function( owner ) { - - // Accepts only: - // - Node - // - Node.ELEMENT_NODE - // - Node.DOCUMENT_NODE - // - Object - // - Any - return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); -}; - - - - -function Data() { - this.expando = jQuery.expando + Data.uid++; -} - -Data.uid = 1; - -Data.prototype = { - - cache: function( owner ) { - - // Check if the owner object already has a cache - var value = owner[ this.expando ]; - - // If not, create one - if ( !value ) { - value = {}; - - // We can accept data for non-element nodes in modern browsers, - // but we should not, see #8335. - // Always return an empty object. - if ( acceptData( owner ) ) { - - // If it is a node unlikely to be stringify-ed or looped over - // use plain assignment - if ( owner.nodeType ) { - owner[ this.expando ] = value; - - // Otherwise secure it in a non-enumerable property - // configurable must be true to allow the property to be - // deleted when data is removed - } else { - Object.defineProperty( owner, this.expando, { - value: value, - configurable: true - } ); - } - } - } - - return value; - }, - set: function( owner, data, value ) { - var prop, - cache = this.cache( owner ); - - // Handle: [ owner, key, value ] args - // Always use camelCase key (gh-2257) - if ( typeof data === "string" ) { - cache[ camelCase( data ) ] = value; - - // Handle: [ owner, { properties } ] args - } else { - - // Copy the properties one-by-one to the cache object - for ( prop in data ) { - cache[ camelCase( prop ) ] = data[ prop ]; - } - } - return cache; - }, - get: function( owner, key ) { - return key === undefined ? - this.cache( owner ) : - - // Always use camelCase key (gh-2257) - owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; - }, - access: function( owner, key, value ) { - - // In cases where either: - // - // 1. No key was specified - // 2. A string key was specified, but no value provided - // - // Take the "read" path and allow the get method to determine - // which value to return, respectively either: - // - // 1. The entire cache object - // 2. The data stored at the key - // - if ( key === undefined || - ( ( key && typeof key === "string" ) && value === undefined ) ) { - - return this.get( owner, key ); - } - - // When the key is not a string, or both a key and value - // are specified, set or extend (existing objects) with either: - // - // 1. An object of properties - // 2. A key and value - // - this.set( owner, key, value ); - - // Since the "set" path can have two possible entry points - // return the expected data based on which path was taken[*] - return value !== undefined ? value : key; - }, - remove: function( owner, key ) { - var i, - cache = owner[ this.expando ]; - - if ( cache === undefined ) { - return; - } - - if ( key !== undefined ) { - - // Support array or space separated string of keys - if ( Array.isArray( key ) ) { - - // If key is an array of keys... - // We always set camelCase keys, so remove that. - key = key.map( camelCase ); - } else { - key = camelCase( key ); - - // If a key with the spaces exists, use it. - // Otherwise, create an array by matching non-whitespace - key = key in cache ? - [ key ] : - ( key.match( rnothtmlwhite ) || [] ); - } - - i = key.length; - - while ( i-- ) { - delete cache[ key[ i ] ]; - } - } - - // Remove the expando if there's no more data - if ( key === undefined || jQuery.isEmptyObject( cache ) ) { - - // Support: Chrome <=35 - 45 - // Webkit & Blink performance suffers when deleting properties - // from DOM nodes, so set to undefined instead - // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) - if ( owner.nodeType ) { - owner[ this.expando ] = undefined; - } else { - delete owner[ this.expando ]; - } - } - }, - hasData: function( owner ) { - var cache = owner[ this.expando ]; - return cache !== undefined && !jQuery.isEmptyObject( cache ); - } -}; -var dataPriv = new Data(); - -var dataUser = new Data(); - - - -// Implementation Summary -// -// 1. Enforce API surface and semantic compatibility with 1.9.x branch -// 2. Improve the module's maintainability by reducing the storage -// paths to a single mechanism. -// 3. Use the same single mechanism to support "private" and "user" data. -// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) -// 5. Avoid exposing implementation details on user objects (eg. expando properties) -// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 - -var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, - rmultiDash = /[A-Z]/g; - -function getData( data ) { - if ( data === "true" ) { - return true; - } - - if ( data === "false" ) { - return false; - } - - if ( data === "null" ) { - return null; - } - - // Only convert to a number if it doesn't change the string - if ( data === +data + "" ) { - return +data; - } - - if ( rbrace.test( data ) ) { - return JSON.parse( data ); - } - - return data; -} - -function dataAttr( elem, key, data ) { - var name; - - // If nothing was found internally, try to fetch any - // data from the HTML5 data-* attribute - if ( data === undefined && elem.nodeType === 1 ) { - name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); - data = elem.getAttribute( name ); - - if ( typeof data === "string" ) { - try { - data = getData( data ); - } catch ( e ) {} - - // Make sure we set the data so it isn't changed later - dataUser.set( elem, key, data ); - } else { - data = undefined; - } - } - return data; -} - -jQuery.extend( { - hasData: function( elem ) { - return dataUser.hasData( elem ) || dataPriv.hasData( elem ); - }, - - data: function( elem, name, data ) { - return dataUser.access( elem, name, data ); - }, - - removeData: function( elem, name ) { - dataUser.remove( elem, name ); - }, - - // TODO: Now that all calls to _data and _removeData have been replaced - // with direct calls to dataPriv methods, these can be deprecated. - _data: function( elem, name, data ) { - return dataPriv.access( elem, name, data ); - }, - - _removeData: function( elem, name ) { - dataPriv.remove( elem, name ); - } -} ); - -jQuery.fn.extend( { - data: function( key, value ) { - var i, name, data, - elem = this[ 0 ], - attrs = elem && elem.attributes; - - // Gets all values - if ( key === undefined ) { - if ( this.length ) { - data = dataUser.get( elem ); - - if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { - i = attrs.length; - while ( i-- ) { - - // Support: IE 11 only - // The attrs elements can be null (#14894) - if ( attrs[ i ] ) { - name = attrs[ i ].name; - if ( name.indexOf( "data-" ) === 0 ) { - name = camelCase( name.slice( 5 ) ); - dataAttr( elem, name, data[ name ] ); - } - } - } - dataPriv.set( elem, "hasDataAttrs", true ); - } - } - - return data; - } - - // Sets multiple values - if ( typeof key === "object" ) { - return this.each( function() { - dataUser.set( this, key ); - } ); - } - - return access( this, function( value ) { - var data; - - // The calling jQuery object (element matches) is not empty - // (and therefore has an element appears at this[ 0 ]) and the - // `value` parameter was not undefined. An empty jQuery object - // will result in `undefined` for elem = this[ 0 ] which will - // throw an exception if an attempt to read a data cache is made. - if ( elem && value === undefined ) { - - // Attempt to get data from the cache - // The key will always be camelCased in Data - data = dataUser.get( elem, key ); - if ( data !== undefined ) { - return data; - } - - // Attempt to "discover" the data in - // HTML5 custom data-* attrs - data = dataAttr( elem, key ); - if ( data !== undefined ) { - return data; - } - - // We tried really hard, but the data doesn't exist. - return; - } - - // Set the data... - this.each( function() { - - // We always store the camelCased key - dataUser.set( this, key, value ); - } ); - }, null, value, arguments.length > 1, null, true ); - }, - - removeData: function( key ) { - return this.each( function() { - dataUser.remove( this, key ); - } ); - } -} ); - - -jQuery.extend( { - queue: function( elem, type, data ) { - var queue; - - if ( elem ) { - type = ( type || "fx" ) + "queue"; - queue = dataPriv.get( elem, type ); - - // Speed up dequeue by getting out quickly if this is just a lookup - if ( data ) { - if ( !queue || Array.isArray( data ) ) { - queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); - } else { - queue.push( data ); - } - } - return queue || []; - } - }, - - dequeue: function( elem, type ) { - type = type || "fx"; - - var queue = jQuery.queue( elem, type ), - startLength = queue.length, - fn = queue.shift(), - hooks = jQuery._queueHooks( elem, type ), - next = function() { - jQuery.dequeue( elem, type ); - }; - - // If the fx queue is dequeued, always remove the progress sentinel - if ( fn === "inprogress" ) { - fn = queue.shift(); - startLength--; - } - - if ( fn ) { - - // Add a progress sentinel to prevent the fx queue from being - // automatically dequeued - if ( type === "fx" ) { - queue.unshift( "inprogress" ); - } - - // Clear up the last queue stop function - delete hooks.stop; - fn.call( elem, next, hooks ); - } - - if ( !startLength && hooks ) { - hooks.empty.fire(); - } - }, - - // Not public - generate a queueHooks object, or return the current one - _queueHooks: function( elem, type ) { - var key = type + "queueHooks"; - return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { - empty: jQuery.Callbacks( "once memory" ).add( function() { - dataPriv.remove( elem, [ type + "queue", key ] ); - } ) - } ); - } -} ); - -jQuery.fn.extend( { - queue: function( type, data ) { - var setter = 2; - - if ( typeof type !== "string" ) { - data = type; - type = "fx"; - setter--; - } - - if ( arguments.length < setter ) { - return jQuery.queue( this[ 0 ], type ); - } - - return data === undefined ? - this : - this.each( function() { - var queue = jQuery.queue( this, type, data ); - - // Ensure a hooks for this queue - jQuery._queueHooks( this, type ); - - if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { - jQuery.dequeue( this, type ); - } - } ); - }, - dequeue: function( type ) { - return this.each( function() { - jQuery.dequeue( this, type ); - } ); - }, - clearQueue: function( type ) { - return this.queue( type || "fx", [] ); - }, - - // Get a promise resolved when queues of a certain type - // are emptied (fx is the type by default) - promise: function( type, obj ) { - var tmp, - count = 1, - defer = jQuery.Deferred(), - elements = this, - i = this.length, - resolve = function() { - if ( !( --count ) ) { - defer.resolveWith( elements, [ elements ] ); - } - }; - - if ( typeof type !== "string" ) { - obj = type; - type = undefined; - } - type = type || "fx"; - - while ( i-- ) { - tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); - if ( tmp && tmp.empty ) { - count++; - tmp.empty.add( resolve ); - } - } - resolve(); - return defer.promise( obj ); - } -} ); -var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; - -var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); - - -var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; - -var documentElement = document.documentElement; - - - - var isAttached = function( elem ) { - return jQuery.contains( elem.ownerDocument, elem ); - }, - composed = { composed: true }; - - // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only - // Check attachment across shadow DOM boundaries when possible (gh-3504) - // Support: iOS 10.0-10.2 only - // Early iOS 10 versions support `attachShadow` but not `getRootNode`, - // leading to errors. We need to check for `getRootNode`. - if ( documentElement.getRootNode ) { - isAttached = function( elem ) { - return jQuery.contains( elem.ownerDocument, elem ) || - elem.getRootNode( composed ) === elem.ownerDocument; - }; - } -var isHiddenWithinTree = function( elem, el ) { - - // isHiddenWithinTree might be called from jQuery#filter function; - // in that case, element will be second argument - elem = el || elem; - - // Inline style trumps all - return elem.style.display === "none" || - elem.style.display === "" && - - // Otherwise, check computed style - // Support: Firefox <=43 - 45 - // Disconnected elements can have computed display: none, so first confirm that elem is - // in the document. - isAttached( elem ) && - - jQuery.css( elem, "display" ) === "none"; - }; - - - -function adjustCSS( elem, prop, valueParts, tween ) { - var adjusted, scale, - maxIterations = 20, - currentValue = tween ? - function() { - return tween.cur(); - } : - function() { - return jQuery.css( elem, prop, "" ); - }, - initial = currentValue(), - unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), - - // Starting value computation is required for potential unit mismatches - initialInUnit = elem.nodeType && - ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && - rcssNum.exec( jQuery.css( elem, prop ) ); - - if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { - - // Support: Firefox <=54 - // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) - initial = initial / 2; - - // Trust units reported by jQuery.css - unit = unit || initialInUnit[ 3 ]; - - // Iteratively approximate from a nonzero starting point - initialInUnit = +initial || 1; - - while ( maxIterations-- ) { - - // Evaluate and update our best guess (doubling guesses that zero out). - // Finish if the scale equals or crosses 1 (making the old*new product non-positive). - jQuery.style( elem, prop, initialInUnit + unit ); - if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { - maxIterations = 0; - } - initialInUnit = initialInUnit / scale; - - } - - initialInUnit = initialInUnit * 2; - jQuery.style( elem, prop, initialInUnit + unit ); - - // Make sure we update the tween properties later on - valueParts = valueParts || []; - } - - if ( valueParts ) { - initialInUnit = +initialInUnit || +initial || 0; - - // Apply relative offset (+=/-=) if specified - adjusted = valueParts[ 1 ] ? - initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : - +valueParts[ 2 ]; - if ( tween ) { - tween.unit = unit; - tween.start = initialInUnit; - tween.end = adjusted; - } - } - return adjusted; -} - - -var defaultDisplayMap = {}; - -function getDefaultDisplay( elem ) { - var temp, - doc = elem.ownerDocument, - nodeName = elem.nodeName, - display = defaultDisplayMap[ nodeName ]; - - if ( display ) { - return display; - } - - temp = doc.body.appendChild( doc.createElement( nodeName ) ); - display = jQuery.css( temp, "display" ); - - temp.parentNode.removeChild( temp ); - - if ( display === "none" ) { - display = "block"; - } - defaultDisplayMap[ nodeName ] = display; - - return display; -} - -function showHide( elements, show ) { - var display, elem, - values = [], - index = 0, - length = elements.length; - - // Determine new display value for elements that need to change - for ( ; index < length; index++ ) { - elem = elements[ index ]; - if ( !elem.style ) { - continue; - } - - display = elem.style.display; - if ( show ) { - - // Since we force visibility upon cascade-hidden elements, an immediate (and slow) - // check is required in this first loop unless we have a nonempty display value (either - // inline or about-to-be-restored) - if ( display === "none" ) { - values[ index ] = dataPriv.get( elem, "display" ) || null; - if ( !values[ index ] ) { - elem.style.display = ""; - } - } - if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { - values[ index ] = getDefaultDisplay( elem ); - } - } else { - if ( display !== "none" ) { - values[ index ] = "none"; - - // Remember what we're overwriting - dataPriv.set( elem, "display", display ); - } - } - } - - // Set the display of the elements in a second loop to avoid constant reflow - for ( index = 0; index < length; index++ ) { - if ( values[ index ] != null ) { - elements[ index ].style.display = values[ index ]; - } - } - - return elements; -} - -jQuery.fn.extend( { - show: function() { - return showHide( this, true ); - }, - hide: function() { - return showHide( this ); - }, - toggle: function( state ) { - if ( typeof state === "boolean" ) { - return state ? this.show() : this.hide(); - } - - return this.each( function() { - if ( isHiddenWithinTree( this ) ) { - jQuery( this ).show(); - } else { - jQuery( this ).hide(); - } - } ); - } -} ); -var rcheckableType = ( /^(?:checkbox|radio)$/i ); - -var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); - -var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); - - - -( function() { - var fragment = document.createDocumentFragment(), - div = fragment.appendChild( document.createElement( "div" ) ), - input = document.createElement( "input" ); - - // Support: Android 4.0 - 4.3 only - // Check state lost if the name is set (#11217) - // Support: Windows Web Apps (WWA) - // `name` and `type` must use .setAttribute for WWA (#14901) - input.setAttribute( "type", "radio" ); - input.setAttribute( "checked", "checked" ); - input.setAttribute( "name", "t" ); - - div.appendChild( input ); - - // Support: Android <=4.1 only - // Older WebKit doesn't clone checked state correctly in fragments - support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; - - // Support: IE <=11 only - // Make sure textarea (and checkbox) defaultValue is properly cloned - div.innerHTML = ""; - support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; - - // Support: IE <=9 only - // IE <=9 replaces "; - support.option = !!div.lastChild; -} )(); - - -// We have to close these tags to support XHTML (#13200) -var wrapMap = { - - // XHTML parsers do not magically insert elements in the - // same way that tag soup parsers do. So we cannot shorten - // this by omitting or other required elements. - thead: [ 1, "", "
" ], - col: [ 2, "", "
" ], - tr: [ 2, "", "
" ], - td: [ 3, "", "
" ], - - _default: [ 0, "", "" ] -}; - -wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; -wrapMap.th = wrapMap.td; - -// Support: IE <=9 only -if ( !support.option ) { - wrapMap.optgroup = wrapMap.option = [ 1, "" ]; -} - - -function getAll( context, tag ) { - - // Support: IE <=9 - 11 only - // Use typeof to avoid zero-argument method invocation on host objects (#15151) - var ret; - - if ( typeof context.getElementsByTagName !== "undefined" ) { - ret = context.getElementsByTagName( tag || "*" ); - - } else if ( typeof context.querySelectorAll !== "undefined" ) { - ret = context.querySelectorAll( tag || "*" ); - - } else { - ret = []; - } - - if ( tag === undefined || tag && nodeName( context, tag ) ) { - return jQuery.merge( [ context ], ret ); - } - - return ret; -} - - -// Mark scripts as having already been evaluated -function setGlobalEval( elems, refElements ) { - var i = 0, - l = elems.length; - - for ( ; i < l; i++ ) { - dataPriv.set( - elems[ i ], - "globalEval", - !refElements || dataPriv.get( refElements[ i ], "globalEval" ) - ); - } -} - - -var rhtml = /<|&#?\w+;/; - -function buildFragment( elems, context, scripts, selection, ignored ) { - var elem, tmp, tag, wrap, attached, j, - fragment = context.createDocumentFragment(), - nodes = [], - i = 0, - l = elems.length; - - for ( ; i < l; i++ ) { - elem = elems[ i ]; - - if ( elem || elem === 0 ) { - - // Add nodes directly - if ( toType( elem ) === "object" ) { - - // Support: Android <=4.0 only, PhantomJS 1 only - // push.apply(_, arraylike) throws on ancient WebKit - jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); - - // Convert non-html into a text node - } else if ( !rhtml.test( elem ) ) { - nodes.push( context.createTextNode( elem ) ); - - // Convert html into DOM nodes - } else { - tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); - - // Deserialize a standard representation - tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); - wrap = wrapMap[ tag ] || wrapMap._default; - tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; - - // Descend through wrappers to the right content - j = wrap[ 0 ]; - while ( j-- ) { - tmp = tmp.lastChild; - } - - // Support: Android <=4.0 only, PhantomJS 1 only - // push.apply(_, arraylike) throws on ancient WebKit - jQuery.merge( nodes, tmp.childNodes ); - - // Remember the top-level container - tmp = fragment.firstChild; - - // Ensure the created nodes are orphaned (#12392) - tmp.textContent = ""; - } - } - } - - // Remove wrapper from fragment - fragment.textContent = ""; - - i = 0; - while ( ( elem = nodes[ i++ ] ) ) { - - // Skip elements already in the context collection (trac-4087) - if ( selection && jQuery.inArray( elem, selection ) > -1 ) { - if ( ignored ) { - ignored.push( elem ); - } - continue; - } - - attached = isAttached( elem ); - - // Append to fragment - tmp = getAll( fragment.appendChild( elem ), "script" ); - - // Preserve script evaluation history - if ( attached ) { - setGlobalEval( tmp ); - } - - // Capture executables - if ( scripts ) { - j = 0; - while ( ( elem = tmp[ j++ ] ) ) { - if ( rscriptType.test( elem.type || "" ) ) { - scripts.push( elem ); - } - } - } - } - - return fragment; -} - - -var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; - -function returnTrue() { - return true; -} - -function returnFalse() { - return false; -} - -// Support: IE <=9 - 11+ -// focus() and blur() are asynchronous, except when they are no-op. -// So expect focus to be synchronous when the element is already active, -// and blur to be synchronous when the element is not already active. -// (focus and blur are always synchronous in other supported browsers, -// this just defines when we can count on it). -function expectSync( elem, type ) { - return ( elem === safeActiveElement() ) === ( type === "focus" ); -} - -// Support: IE <=9 only -// Accessing document.activeElement can throw unexpectedly -// https://bugs.jquery.com/ticket/13393 -function safeActiveElement() { - try { - return document.activeElement; - } catch ( err ) { } -} - -function on( elem, types, selector, data, fn, one ) { - var origFn, type; - - // Types can be a map of types/handlers - if ( typeof types === "object" ) { - - // ( types-Object, selector, data ) - if ( typeof selector !== "string" ) { - - // ( types-Object, data ) - data = data || selector; - selector = undefined; - } - for ( type in types ) { - on( elem, type, selector, data, types[ type ], one ); - } - return elem; - } - - if ( data == null && fn == null ) { - - // ( types, fn ) - fn = selector; - data = selector = undefined; - } else if ( fn == null ) { - if ( typeof selector === "string" ) { - - // ( types, selector, fn ) - fn = data; - data = undefined; - } else { - - // ( types, data, fn ) - fn = data; - data = selector; - selector = undefined; - } - } - if ( fn === false ) { - fn = returnFalse; - } else if ( !fn ) { - return elem; - } - - if ( one === 1 ) { - origFn = fn; - fn = function( event ) { - - // Can use an empty set, since event contains the info - jQuery().off( event ); - return origFn.apply( this, arguments ); - }; - - // Use same guid so caller can remove using origFn - fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); - } - return elem.each( function() { - jQuery.event.add( this, types, fn, data, selector ); - } ); -} - -/* - * Helper functions for managing events -- not part of the public interface. - * Props to Dean Edwards' addEvent library for many of the ideas. - */ -jQuery.event = { - - global: {}, - - add: function( elem, types, handler, data, selector ) { - - var handleObjIn, eventHandle, tmp, - events, t, handleObj, - special, handlers, type, namespaces, origType, - elemData = dataPriv.get( elem ); - - // Only attach events to objects that accept data - if ( !acceptData( elem ) ) { - return; - } - - // Caller can pass in an object of custom data in lieu of the handler - if ( handler.handler ) { - handleObjIn = handler; - handler = handleObjIn.handler; - selector = handleObjIn.selector; - } - - // Ensure that invalid selectors throw exceptions at attach time - // Evaluate against documentElement in case elem is a non-element node (e.g., document) - if ( selector ) { - jQuery.find.matchesSelector( documentElement, selector ); - } - - // Make sure that the handler has a unique ID, used to find/remove it later - if ( !handler.guid ) { - handler.guid = jQuery.guid++; - } - - // Init the element's event structure and main handler, if this is the first - if ( !( events = elemData.events ) ) { - events = elemData.events = Object.create( null ); - } - if ( !( eventHandle = elemData.handle ) ) { - eventHandle = elemData.handle = function( e ) { - - // Discard the second event of a jQuery.event.trigger() and - // when an event is called after a page has unloaded - return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? - jQuery.event.dispatch.apply( elem, arguments ) : undefined; - }; - } - - // Handle multiple events separated by a space - types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; - t = types.length; - while ( t-- ) { - tmp = rtypenamespace.exec( types[ t ] ) || []; - type = origType = tmp[ 1 ]; - namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); - - // There *must* be a type, no attaching namespace-only handlers - if ( !type ) { - continue; - } - - // If event changes its type, use the special event handlers for the changed type - special = jQuery.event.special[ type ] || {}; - - // If selector defined, determine special event api type, otherwise given type - type = ( selector ? special.delegateType : special.bindType ) || type; - - // Update special based on newly reset type - special = jQuery.event.special[ type ] || {}; - - // handleObj is passed to all event handlers - handleObj = jQuery.extend( { - type: type, - origType: origType, - data: data, - handler: handler, - guid: handler.guid, - selector: selector, - needsContext: selector && jQuery.expr.match.needsContext.test( selector ), - namespace: namespaces.join( "." ) - }, handleObjIn ); - - // Init the event handler queue if we're the first - if ( !( handlers = events[ type ] ) ) { - handlers = events[ type ] = []; - handlers.delegateCount = 0; - - // Only use addEventListener if the special events handler returns false - if ( !special.setup || - special.setup.call( elem, data, namespaces, eventHandle ) === false ) { - - if ( elem.addEventListener ) { - elem.addEventListener( type, eventHandle ); - } - } - } - - if ( special.add ) { - special.add.call( elem, handleObj ); - - if ( !handleObj.handler.guid ) { - handleObj.handler.guid = handler.guid; - } - } - - // Add to the element's handler list, delegates in front - if ( selector ) { - handlers.splice( handlers.delegateCount++, 0, handleObj ); - } else { - handlers.push( handleObj ); - } - - // Keep track of which events have ever been used, for event optimization - jQuery.event.global[ type ] = true; - } - - }, - - // Detach an event or set of events from an element - remove: function( elem, types, handler, selector, mappedTypes ) { - - var j, origCount, tmp, - events, t, handleObj, - special, handlers, type, namespaces, origType, - elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); - - if ( !elemData || !( events = elemData.events ) ) { - return; - } - - // Once for each type.namespace in types; type may be omitted - types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; - t = types.length; - while ( t-- ) { - tmp = rtypenamespace.exec( types[ t ] ) || []; - type = origType = tmp[ 1 ]; - namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); - - // Unbind all events (on this namespace, if provided) for the element - if ( !type ) { - for ( type in events ) { - jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); - } - continue; - } - - special = jQuery.event.special[ type ] || {}; - type = ( selector ? special.delegateType : special.bindType ) || type; - handlers = events[ type ] || []; - tmp = tmp[ 2 ] && - new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); - - // Remove matching events - origCount = j = handlers.length; - while ( j-- ) { - handleObj = handlers[ j ]; - - if ( ( mappedTypes || origType === handleObj.origType ) && - ( !handler || handler.guid === handleObj.guid ) && - ( !tmp || tmp.test( handleObj.namespace ) ) && - ( !selector || selector === handleObj.selector || - selector === "**" && handleObj.selector ) ) { - handlers.splice( j, 1 ); - - if ( handleObj.selector ) { - handlers.delegateCount--; - } - if ( special.remove ) { - special.remove.call( elem, handleObj ); - } - } - } - - // Remove generic event handler if we removed something and no more handlers exist - // (avoids potential for endless recursion during removal of special event handlers) - if ( origCount && !handlers.length ) { - if ( !special.teardown || - special.teardown.call( elem, namespaces, elemData.handle ) === false ) { - - jQuery.removeEvent( elem, type, elemData.handle ); - } - - delete events[ type ]; - } - } - - // Remove data and the expando if it's no longer used - if ( jQuery.isEmptyObject( events ) ) { - dataPriv.remove( elem, "handle events" ); - } - }, - - dispatch: function( nativeEvent ) { - - var i, j, ret, matched, handleObj, handlerQueue, - args = new Array( arguments.length ), - - // Make a writable jQuery.Event from the native event object - event = jQuery.event.fix( nativeEvent ), - - handlers = ( - dataPriv.get( this, "events" ) || Object.create( null ) - )[ event.type ] || [], - special = jQuery.event.special[ event.type ] || {}; - - // Use the fix-ed jQuery.Event rather than the (read-only) native event - args[ 0 ] = event; - - for ( i = 1; i < arguments.length; i++ ) { - args[ i ] = arguments[ i ]; - } - - event.delegateTarget = this; - - // Call the preDispatch hook for the mapped type, and let it bail if desired - if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { - return; - } - - // Determine handlers - handlerQueue = jQuery.event.handlers.call( this, event, handlers ); - - // Run delegates first; they may want to stop propagation beneath us - i = 0; - while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { - event.currentTarget = matched.elem; - - j = 0; - while ( ( handleObj = matched.handlers[ j++ ] ) && - !event.isImmediatePropagationStopped() ) { - - // If the event is namespaced, then each handler is only invoked if it is - // specially universal or its namespaces are a superset of the event's. - if ( !event.rnamespace || handleObj.namespace === false || - event.rnamespace.test( handleObj.namespace ) ) { - - event.handleObj = handleObj; - event.data = handleObj.data; - - ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || - handleObj.handler ).apply( matched.elem, args ); - - if ( ret !== undefined ) { - if ( ( event.result = ret ) === false ) { - event.preventDefault(); - event.stopPropagation(); - } - } - } - } - } - - // Call the postDispatch hook for the mapped type - if ( special.postDispatch ) { - special.postDispatch.call( this, event ); - } - - return event.result; - }, - - handlers: function( event, handlers ) { - var i, handleObj, sel, matchedHandlers, matchedSelectors, - handlerQueue = [], - delegateCount = handlers.delegateCount, - cur = event.target; - - // Find delegate handlers - if ( delegateCount && - - // Support: IE <=9 - // Black-hole SVG instance trees (trac-13180) - cur.nodeType && - - // Support: Firefox <=42 - // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) - // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click - // Support: IE 11 only - // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) - !( event.type === "click" && event.button >= 1 ) ) { - - for ( ; cur !== this; cur = cur.parentNode || this ) { - - // Don't check non-elements (#13208) - // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) - if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { - matchedHandlers = []; - matchedSelectors = {}; - for ( i = 0; i < delegateCount; i++ ) { - handleObj = handlers[ i ]; - - // Don't conflict with Object.prototype properties (#13203) - sel = handleObj.selector + " "; - - if ( matchedSelectors[ sel ] === undefined ) { - matchedSelectors[ sel ] = handleObj.needsContext ? - jQuery( sel, this ).index( cur ) > -1 : - jQuery.find( sel, this, null, [ cur ] ).length; - } - if ( matchedSelectors[ sel ] ) { - matchedHandlers.push( handleObj ); - } - } - if ( matchedHandlers.length ) { - handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); - } - } - } - } - - // Add the remaining (directly-bound) handlers - cur = this; - if ( delegateCount < handlers.length ) { - handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); - } - - return handlerQueue; - }, - - addProp: function( name, hook ) { - Object.defineProperty( jQuery.Event.prototype, name, { - enumerable: true, - configurable: true, - - get: isFunction( hook ) ? - function() { - if ( this.originalEvent ) { - return hook( this.originalEvent ); - } - } : - function() { - if ( this.originalEvent ) { - return this.originalEvent[ name ]; - } - }, - - set: function( value ) { - Object.defineProperty( this, name, { - enumerable: true, - configurable: true, - writable: true, - value: value - } ); - } - } ); - }, - - fix: function( originalEvent ) { - return originalEvent[ jQuery.expando ] ? - originalEvent : - new jQuery.Event( originalEvent ); - }, - - special: { - load: { - - // Prevent triggered image.load events from bubbling to window.load - noBubble: true - }, - click: { - - // Utilize native event to ensure correct state for checkable inputs - setup: function( data ) { - - // For mutual compressibility with _default, replace `this` access with a local var. - // `|| data` is dead code meant only to preserve the variable through minification. - var el = this || data; - - // Claim the first handler - if ( rcheckableType.test( el.type ) && - el.click && nodeName( el, "input" ) ) { - - // dataPriv.set( el, "click", ... ) - leverageNative( el, "click", returnTrue ); - } - - // Return false to allow normal processing in the caller - return false; - }, - trigger: function( data ) { - - // For mutual compressibility with _default, replace `this` access with a local var. - // `|| data` is dead code meant only to preserve the variable through minification. - var el = this || data; - - // Force setup before triggering a click - if ( rcheckableType.test( el.type ) && - el.click && nodeName( el, "input" ) ) { - - leverageNative( el, "click" ); - } - - // Return non-false to allow normal event-path propagation - return true; - }, - - // For cross-browser consistency, suppress native .click() on links - // Also prevent it if we're currently inside a leveraged native-event stack - _default: function( event ) { - var target = event.target; - return rcheckableType.test( target.type ) && - target.click && nodeName( target, "input" ) && - dataPriv.get( target, "click" ) || - nodeName( target, "a" ); - } - }, - - beforeunload: { - postDispatch: function( event ) { - - // Support: Firefox 20+ - // Firefox doesn't alert if the returnValue field is not set. - if ( event.result !== undefined && event.originalEvent ) { - event.originalEvent.returnValue = event.result; - } - } - } - } -}; - -// Ensure the presence of an event listener that handles manually-triggered -// synthetic events by interrupting progress until reinvoked in response to -// *native* events that it fires directly, ensuring that state changes have -// already occurred before other listeners are invoked. -function leverageNative( el, type, expectSync ) { - - // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add - if ( !expectSync ) { - if ( dataPriv.get( el, type ) === undefined ) { - jQuery.event.add( el, type, returnTrue ); - } - return; - } - - // Register the controller as a special universal handler for all event namespaces - dataPriv.set( el, type, false ); - jQuery.event.add( el, type, { - namespace: false, - handler: function( event ) { - var notAsync, result, - saved = dataPriv.get( this, type ); - - if ( ( event.isTrigger & 1 ) && this[ type ] ) { - - // Interrupt processing of the outer synthetic .trigger()ed event - // Saved data should be false in such cases, but might be a leftover capture object - // from an async native handler (gh-4350) - if ( !saved.length ) { - - // Store arguments for use when handling the inner native event - // There will always be at least one argument (an event object), so this array - // will not be confused with a leftover capture object. - saved = slice.call( arguments ); - dataPriv.set( this, type, saved ); - - // Trigger the native event and capture its result - // Support: IE <=9 - 11+ - // focus() and blur() are asynchronous - notAsync = expectSync( this, type ); - this[ type ](); - result = dataPriv.get( this, type ); - if ( saved !== result || notAsync ) { - dataPriv.set( this, type, false ); - } else { - result = {}; - } - if ( saved !== result ) { - - // Cancel the outer synthetic event - event.stopImmediatePropagation(); - event.preventDefault(); - - // Support: Chrome 86+ - // In Chrome, if an element having a focusout handler is blurred by - // clicking outside of it, it invokes the handler synchronously. If - // that handler calls `.remove()` on the element, the data is cleared, - // leaving `result` undefined. We need to guard against this. - return result && result.value; - } - - // If this is an inner synthetic event for an event with a bubbling surrogate - // (focus or blur), assume that the surrogate already propagated from triggering the - // native event and prevent that from happening again here. - // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the - // bubbling surrogate propagates *after* the non-bubbling base), but that seems - // less bad than duplication. - } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { - event.stopPropagation(); - } - - // If this is a native event triggered above, everything is now in order - // Fire an inner synthetic event with the original arguments - } else if ( saved.length ) { - - // ...and capture the result - dataPriv.set( this, type, { - value: jQuery.event.trigger( - - // Support: IE <=9 - 11+ - // Extend with the prototype to reset the above stopImmediatePropagation() - jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), - saved.slice( 1 ), - this - ) - } ); - - // Abort handling of the native event - event.stopImmediatePropagation(); - } - } - } ); -} - -jQuery.removeEvent = function( elem, type, handle ) { - - // This "if" is needed for plain objects - if ( elem.removeEventListener ) { - elem.removeEventListener( type, handle ); - } -}; - -jQuery.Event = function( src, props ) { - - // Allow instantiation without the 'new' keyword - if ( !( this instanceof jQuery.Event ) ) { - return new jQuery.Event( src, props ); - } - - // Event object - if ( src && src.type ) { - this.originalEvent = src; - this.type = src.type; - - // Events bubbling up the document may have been marked as prevented - // by a handler lower down the tree; reflect the correct value. - this.isDefaultPrevented = src.defaultPrevented || - src.defaultPrevented === undefined && - - // Support: Android <=2.3 only - src.returnValue === false ? - returnTrue : - returnFalse; - - // Create target properties - // Support: Safari <=6 - 7 only - // Target should not be a text node (#504, #13143) - this.target = ( src.target && src.target.nodeType === 3 ) ? - src.target.parentNode : - src.target; - - this.currentTarget = src.currentTarget; - this.relatedTarget = src.relatedTarget; - - // Event type - } else { - this.type = src; - } - - // Put explicitly provided properties onto the event object - if ( props ) { - jQuery.extend( this, props ); - } - - // Create a timestamp if incoming event doesn't have one - this.timeStamp = src && src.timeStamp || Date.now(); - - // Mark it as fixed - this[ jQuery.expando ] = true; -}; - -// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding -// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html -jQuery.Event.prototype = { - constructor: jQuery.Event, - isDefaultPrevented: returnFalse, - isPropagationStopped: returnFalse, - isImmediatePropagationStopped: returnFalse, - isSimulated: false, - - preventDefault: function() { - var e = this.originalEvent; - - this.isDefaultPrevented = returnTrue; - - if ( e && !this.isSimulated ) { - e.preventDefault(); - } - }, - stopPropagation: function() { - var e = this.originalEvent; - - this.isPropagationStopped = returnTrue; - - if ( e && !this.isSimulated ) { - e.stopPropagation(); - } - }, - stopImmediatePropagation: function() { - var e = this.originalEvent; - - this.isImmediatePropagationStopped = returnTrue; - - if ( e && !this.isSimulated ) { - e.stopImmediatePropagation(); - } - - this.stopPropagation(); - } -}; - -// Includes all common event props including KeyEvent and MouseEvent specific props -jQuery.each( { - altKey: true, - bubbles: true, - cancelable: true, - changedTouches: true, - ctrlKey: true, - detail: true, - eventPhase: true, - metaKey: true, - pageX: true, - pageY: true, - shiftKey: true, - view: true, - "char": true, - code: true, - charCode: true, - key: true, - keyCode: true, - button: true, - buttons: true, - clientX: true, - clientY: true, - offsetX: true, - offsetY: true, - pointerId: true, - pointerType: true, - screenX: true, - screenY: true, - targetTouches: true, - toElement: true, - touches: true, - which: true -}, jQuery.event.addProp ); - -jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { - jQuery.event.special[ type ] = { - - // Utilize native event if possible so blur/focus sequence is correct - setup: function() { - - // Claim the first handler - // dataPriv.set( this, "focus", ... ) - // dataPriv.set( this, "blur", ... ) - leverageNative( this, type, expectSync ); - - // Return false to allow normal processing in the caller - return false; - }, - trigger: function() { - - // Force setup before trigger - leverageNative( this, type ); - - // Return non-false to allow normal event-path propagation - return true; - }, - - // Suppress native focus or blur as it's already being fired - // in leverageNative. - _default: function() { - return true; - }, - - delegateType: delegateType - }; -} ); - -// Create mouseenter/leave events using mouseover/out and event-time checks -// so that event delegation works in jQuery. -// Do the same for pointerenter/pointerleave and pointerover/pointerout -// -// Support: Safari 7 only -// Safari sends mouseenter too often; see: -// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 -// for the description of the bug (it existed in older Chrome versions as well). -jQuery.each( { - mouseenter: "mouseover", - mouseleave: "mouseout", - pointerenter: "pointerover", - pointerleave: "pointerout" -}, function( orig, fix ) { - jQuery.event.special[ orig ] = { - delegateType: fix, - bindType: fix, - - handle: function( event ) { - var ret, - target = this, - related = event.relatedTarget, - handleObj = event.handleObj; - - // For mouseenter/leave call the handler if related is outside the target. - // NB: No relatedTarget if the mouse left/entered the browser window - if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { - event.type = handleObj.origType; - ret = handleObj.handler.apply( this, arguments ); - event.type = fix; - } - return ret; - } - }; -} ); - -jQuery.fn.extend( { - - on: function( types, selector, data, fn ) { - return on( this, types, selector, data, fn ); - }, - one: function( types, selector, data, fn ) { - return on( this, types, selector, data, fn, 1 ); - }, - off: function( types, selector, fn ) { - var handleObj, type; - if ( types && types.preventDefault && types.handleObj ) { - - // ( event ) dispatched jQuery.Event - handleObj = types.handleObj; - jQuery( types.delegateTarget ).off( - handleObj.namespace ? - handleObj.origType + "." + handleObj.namespace : - handleObj.origType, - handleObj.selector, - handleObj.handler - ); - return this; - } - if ( typeof types === "object" ) { - - // ( types-object [, selector] ) - for ( type in types ) { - this.off( type, selector, types[ type ] ); - } - return this; - } - if ( selector === false || typeof selector === "function" ) { - - // ( types [, fn] ) - fn = selector; - selector = undefined; - } - if ( fn === false ) { - fn = returnFalse; - } - return this.each( function() { - jQuery.event.remove( this, types, fn, selector ); - } ); - } -} ); - - -var - - // Support: IE <=10 - 11, Edge 12 - 13 only - // In IE/Edge using regex groups here causes severe slowdowns. - // See https://connect.microsoft.com/IE/feedback/details/1736512/ - rnoInnerhtml = /\s*$/g; - -// Prefer a tbody over its parent table for containing new rows -function manipulationTarget( elem, content ) { - if ( nodeName( elem, "table" ) && - nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { - - return jQuery( elem ).children( "tbody" )[ 0 ] || elem; - } - - return elem; -} - -// Replace/restore the type attribute of script elements for safe DOM manipulation -function disableScript( elem ) { - elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; - return elem; -} -function restoreScript( elem ) { - if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { - elem.type = elem.type.slice( 5 ); - } else { - elem.removeAttribute( "type" ); - } - - return elem; -} - -function cloneCopyEvent( src, dest ) { - var i, l, type, pdataOld, udataOld, udataCur, events; - - if ( dest.nodeType !== 1 ) { - return; - } - - // 1. Copy private data: events, handlers, etc. - if ( dataPriv.hasData( src ) ) { - pdataOld = dataPriv.get( src ); - events = pdataOld.events; - - if ( events ) { - dataPriv.remove( dest, "handle events" ); - - for ( type in events ) { - for ( i = 0, l = events[ type ].length; i < l; i++ ) { - jQuery.event.add( dest, type, events[ type ][ i ] ); - } - } - } - } - - // 2. Copy user data - if ( dataUser.hasData( src ) ) { - udataOld = dataUser.access( src ); - udataCur = jQuery.extend( {}, udataOld ); - - dataUser.set( dest, udataCur ); - } -} - -// Fix IE bugs, see support tests -function fixInput( src, dest ) { - var nodeName = dest.nodeName.toLowerCase(); - - // Fails to persist the checked state of a cloned checkbox or radio button. - if ( nodeName === "input" && rcheckableType.test( src.type ) ) { - dest.checked = src.checked; - - // Fails to return the selected option to the default selected state when cloning options - } else if ( nodeName === "input" || nodeName === "textarea" ) { - dest.defaultValue = src.defaultValue; - } -} - -function domManip( collection, args, callback, ignored ) { - - // Flatten any nested arrays - args = flat( args ); - - var fragment, first, scripts, hasScripts, node, doc, - i = 0, - l = collection.length, - iNoClone = l - 1, - value = args[ 0 ], - valueIsFunction = isFunction( value ); - - // We can't cloneNode fragments that contain checked, in WebKit - if ( valueIsFunction || - ( l > 1 && typeof value === "string" && - !support.checkClone && rchecked.test( value ) ) ) { - return collection.each( function( index ) { - var self = collection.eq( index ); - if ( valueIsFunction ) { - args[ 0 ] = value.call( this, index, self.html() ); - } - domManip( self, args, callback, ignored ); - } ); - } - - if ( l ) { - fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); - first = fragment.firstChild; - - if ( fragment.childNodes.length === 1 ) { - fragment = first; - } - - // Require either new content or an interest in ignored elements to invoke the callback - if ( first || ignored ) { - scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); - hasScripts = scripts.length; - - // Use the original fragment for the last item - // instead of the first because it can end up - // being emptied incorrectly in certain situations (#8070). - for ( ; i < l; i++ ) { - node = fragment; - - if ( i !== iNoClone ) { - node = jQuery.clone( node, true, true ); - - // Keep references to cloned scripts for later restoration - if ( hasScripts ) { - - // Support: Android <=4.0 only, PhantomJS 1 only - // push.apply(_, arraylike) throws on ancient WebKit - jQuery.merge( scripts, getAll( node, "script" ) ); - } - } - - callback.call( collection[ i ], node, i ); - } - - if ( hasScripts ) { - doc = scripts[ scripts.length - 1 ].ownerDocument; - - // Reenable scripts - jQuery.map( scripts, restoreScript ); - - // Evaluate executable scripts on first document insertion - for ( i = 0; i < hasScripts; i++ ) { - node = scripts[ i ]; - if ( rscriptType.test( node.type || "" ) && - !dataPriv.access( node, "globalEval" ) && - jQuery.contains( doc, node ) ) { - - if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { - - // Optional AJAX dependency, but won't run scripts if not present - if ( jQuery._evalUrl && !node.noModule ) { - jQuery._evalUrl( node.src, { - nonce: node.nonce || node.getAttribute( "nonce" ) - }, doc ); - } - } else { - DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); - } - } - } - } - } - } - - return collection; -} - -function remove( elem, selector, keepData ) { - var node, - nodes = selector ? jQuery.filter( selector, elem ) : elem, - i = 0; - - for ( ; ( node = nodes[ i ] ) != null; i++ ) { - if ( !keepData && node.nodeType === 1 ) { - jQuery.cleanData( getAll( node ) ); - } - - if ( node.parentNode ) { - if ( keepData && isAttached( node ) ) { - setGlobalEval( getAll( node, "script" ) ); - } - node.parentNode.removeChild( node ); - } - } - - return elem; -} - -jQuery.extend( { - htmlPrefilter: function( html ) { - return html; - }, - - clone: function( elem, dataAndEvents, deepDataAndEvents ) { - var i, l, srcElements, destElements, - clone = elem.cloneNode( true ), - inPage = isAttached( elem ); - - // Fix IE cloning issues - if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && - !jQuery.isXMLDoc( elem ) ) { - - // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 - destElements = getAll( clone ); - srcElements = getAll( elem ); - - for ( i = 0, l = srcElements.length; i < l; i++ ) { - fixInput( srcElements[ i ], destElements[ i ] ); - } - } - - // Copy the events from the original to the clone - if ( dataAndEvents ) { - if ( deepDataAndEvents ) { - srcElements = srcElements || getAll( elem ); - destElements = destElements || getAll( clone ); - - for ( i = 0, l = srcElements.length; i < l; i++ ) { - cloneCopyEvent( srcElements[ i ], destElements[ i ] ); - } - } else { - cloneCopyEvent( elem, clone ); - } - } - - // Preserve script evaluation history - destElements = getAll( clone, "script" ); - if ( destElements.length > 0 ) { - setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); - } - - // Return the cloned set - return clone; - }, - - cleanData: function( elems ) { - var data, elem, type, - special = jQuery.event.special, - i = 0; - - for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { - if ( acceptData( elem ) ) { - if ( ( data = elem[ dataPriv.expando ] ) ) { - if ( data.events ) { - for ( type in data.events ) { - if ( special[ type ] ) { - jQuery.event.remove( elem, type ); - - // This is a shortcut to avoid jQuery.event.remove's overhead - } else { - jQuery.removeEvent( elem, type, data.handle ); - } - } - } - - // Support: Chrome <=35 - 45+ - // Assign undefined instead of using delete, see Data#remove - elem[ dataPriv.expando ] = undefined; - } - if ( elem[ dataUser.expando ] ) { - - // Support: Chrome <=35 - 45+ - // Assign undefined instead of using delete, see Data#remove - elem[ dataUser.expando ] = undefined; - } - } - } - } -} ); - -jQuery.fn.extend( { - detach: function( selector ) { - return remove( this, selector, true ); - }, - - remove: function( selector ) { - return remove( this, selector ); - }, - - text: function( value ) { - return access( this, function( value ) { - return value === undefined ? - jQuery.text( this ) : - this.empty().each( function() { - if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { - this.textContent = value; - } - } ); - }, null, value, arguments.length ); - }, - - append: function() { - return domManip( this, arguments, function( elem ) { - if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { - var target = manipulationTarget( this, elem ); - target.appendChild( elem ); - } - } ); - }, - - prepend: function() { - return domManip( this, arguments, function( elem ) { - if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { - var target = manipulationTarget( this, elem ); - target.insertBefore( elem, target.firstChild ); - } - } ); - }, - - before: function() { - return domManip( this, arguments, function( elem ) { - if ( this.parentNode ) { - this.parentNode.insertBefore( elem, this ); - } - } ); - }, - - after: function() { - return domManip( this, arguments, function( elem ) { - if ( this.parentNode ) { - this.parentNode.insertBefore( elem, this.nextSibling ); - } - } ); - }, - - empty: function() { - var elem, - i = 0; - - for ( ; ( elem = this[ i ] ) != null; i++ ) { - if ( elem.nodeType === 1 ) { - - // Prevent memory leaks - jQuery.cleanData( getAll( elem, false ) ); - - // Remove any remaining nodes - elem.textContent = ""; - } - } - - return this; - }, - - clone: function( dataAndEvents, deepDataAndEvents ) { - dataAndEvents = dataAndEvents == null ? false : dataAndEvents; - deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; - - return this.map( function() { - return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); - } ); - }, - - html: function( value ) { - return access( this, function( value ) { - var elem = this[ 0 ] || {}, - i = 0, - l = this.length; - - if ( value === undefined && elem.nodeType === 1 ) { - return elem.innerHTML; - } - - // See if we can take a shortcut and just use innerHTML - if ( typeof value === "string" && !rnoInnerhtml.test( value ) && - !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { - - value = jQuery.htmlPrefilter( value ); - - try { - for ( ; i < l; i++ ) { - elem = this[ i ] || {}; - - // Remove element nodes and prevent memory leaks - if ( elem.nodeType === 1 ) { - jQuery.cleanData( getAll( elem, false ) ); - elem.innerHTML = value; - } - } - - elem = 0; - - // If using innerHTML throws an exception, use the fallback method - } catch ( e ) {} - } - - if ( elem ) { - this.empty().append( value ); - } - }, null, value, arguments.length ); - }, - - replaceWith: function() { - var ignored = []; - - // Make the changes, replacing each non-ignored context element with the new content - return domManip( this, arguments, function( elem ) { - var parent = this.parentNode; - - if ( jQuery.inArray( this, ignored ) < 0 ) { - jQuery.cleanData( getAll( this ) ); - if ( parent ) { - parent.replaceChild( elem, this ); - } - } - - // Force callback invocation - }, ignored ); - } -} ); - -jQuery.each( { - appendTo: "append", - prependTo: "prepend", - insertBefore: "before", - insertAfter: "after", - replaceAll: "replaceWith" -}, function( name, original ) { - jQuery.fn[ name ] = function( selector ) { - var elems, - ret = [], - insert = jQuery( selector ), - last = insert.length - 1, - i = 0; - - for ( ; i <= last; i++ ) { - elems = i === last ? this : this.clone( true ); - jQuery( insert[ i ] )[ original ]( elems ); - - // Support: Android <=4.0 only, PhantomJS 1 only - // .get() because push.apply(_, arraylike) throws on ancient WebKit - push.apply( ret, elems.get() ); - } - - return this.pushStack( ret ); - }; -} ); -var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); - -var getStyles = function( elem ) { - - // Support: IE <=11 only, Firefox <=30 (#15098, #14150) - // IE throws on elements created in popups - // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" - var view = elem.ownerDocument.defaultView; - - if ( !view || !view.opener ) { - view = window; - } - - return view.getComputedStyle( elem ); - }; - -var swap = function( elem, options, callback ) { - var ret, name, - old = {}; - - // Remember the old values, and insert the new ones - for ( name in options ) { - old[ name ] = elem.style[ name ]; - elem.style[ name ] = options[ name ]; - } - - ret = callback.call( elem ); - - // Revert the old values - for ( name in options ) { - elem.style[ name ] = old[ name ]; - } - - return ret; -}; - - -var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); - - - -( function() { - - // Executing both pixelPosition & boxSizingReliable tests require only one layout - // so they're executed at the same time to save the second computation. - function computeStyleTests() { - - // This is a singleton, we need to execute it only once - if ( !div ) { - return; - } - - container.style.cssText = "position:absolute;left:-11111px;width:60px;" + - "margin-top:1px;padding:0;border:0"; - div.style.cssText = - "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + - "margin:auto;border:1px;padding:1px;" + - "width:60%;top:1%"; - documentElement.appendChild( container ).appendChild( div ); - - var divStyle = window.getComputedStyle( div ); - pixelPositionVal = divStyle.top !== "1%"; - - // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 - reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; - - // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 - // Some styles come back with percentage values, even though they shouldn't - div.style.right = "60%"; - pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; - - // Support: IE 9 - 11 only - // Detect misreporting of content dimensions for box-sizing:border-box elements - boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; - - // Support: IE 9 only - // Detect overflow:scroll screwiness (gh-3699) - // Support: Chrome <=64 - // Don't get tricked when zoom affects offsetWidth (gh-4029) - div.style.position = "absolute"; - scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; - - documentElement.removeChild( container ); - - // Nullify the div so it wouldn't be stored in the memory and - // it will also be a sign that checks already performed - div = null; - } - - function roundPixelMeasures( measure ) { - return Math.round( parseFloat( measure ) ); - } - - var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, - reliableTrDimensionsVal, reliableMarginLeftVal, - container = document.createElement( "div" ), - div = document.createElement( "div" ); - - // Finish early in limited (non-browser) environments - if ( !div.style ) { - return; - } - - // Support: IE <=9 - 11 only - // Style of cloned element affects source element cloned (#8908) - div.style.backgroundClip = "content-box"; - div.cloneNode( true ).style.backgroundClip = ""; - support.clearCloneStyle = div.style.backgroundClip === "content-box"; - - jQuery.extend( support, { - boxSizingReliable: function() { - computeStyleTests(); - return boxSizingReliableVal; - }, - pixelBoxStyles: function() { - computeStyleTests(); - return pixelBoxStylesVal; - }, - pixelPosition: function() { - computeStyleTests(); - return pixelPositionVal; - }, - reliableMarginLeft: function() { - computeStyleTests(); - return reliableMarginLeftVal; - }, - scrollboxSize: function() { - computeStyleTests(); - return scrollboxSizeVal; - }, - - // Support: IE 9 - 11+, Edge 15 - 18+ - // IE/Edge misreport `getComputedStyle` of table rows with width/height - // set in CSS while `offset*` properties report correct values. - // Behavior in IE 9 is more subtle than in newer versions & it passes - // some versions of this test; make sure not to make it pass there! - // - // Support: Firefox 70+ - // Only Firefox includes border widths - // in computed dimensions. (gh-4529) - reliableTrDimensions: function() { - var table, tr, trChild, trStyle; - if ( reliableTrDimensionsVal == null ) { - table = document.createElement( "table" ); - tr = document.createElement( "tr" ); - trChild = document.createElement( "div" ); - - table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; - tr.style.cssText = "border:1px solid"; - - // Support: Chrome 86+ - // Height set through cssText does not get applied. - // Computed height then comes back as 0. - tr.style.height = "1px"; - trChild.style.height = "9px"; - - // Support: Android 8 Chrome 86+ - // In our bodyBackground.html iframe, - // display for all div elements is set to "inline", - // which causes a problem only in Android 8 Chrome 86. - // Ensuring the div is display: block - // gets around this issue. - trChild.style.display = "block"; - - documentElement - .appendChild( table ) - .appendChild( tr ) - .appendChild( trChild ); - - trStyle = window.getComputedStyle( tr ); - reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + - parseInt( trStyle.borderTopWidth, 10 ) + - parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; - - documentElement.removeChild( table ); - } - return reliableTrDimensionsVal; - } - } ); -} )(); - - -function curCSS( elem, name, computed ) { - var width, minWidth, maxWidth, ret, - - // Support: Firefox 51+ - // Retrieving style before computed somehow - // fixes an issue with getting wrong values - // on detached elements - style = elem.style; - - computed = computed || getStyles( elem ); - - // getPropertyValue is needed for: - // .css('filter') (IE 9 only, #12537) - // .css('--customProperty) (#3144) - if ( computed ) { - ret = computed.getPropertyValue( name ) || computed[ name ]; - - if ( ret === "" && !isAttached( elem ) ) { - ret = jQuery.style( elem, name ); - } - - // A tribute to the "awesome hack by Dean Edwards" - // Android Browser returns percentage for some values, - // but width seems to be reliably pixels. - // This is against the CSSOM draft spec: - // https://drafts.csswg.org/cssom/#resolved-values - if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { - - // Remember the original values - width = style.width; - minWidth = style.minWidth; - maxWidth = style.maxWidth; - - // Put in the new values to get a computed value out - style.minWidth = style.maxWidth = style.width = ret; - ret = computed.width; - - // Revert the changed values - style.width = width; - style.minWidth = minWidth; - style.maxWidth = maxWidth; - } - } - - return ret !== undefined ? - - // Support: IE <=9 - 11 only - // IE returns zIndex value as an integer. - ret + "" : - ret; -} - - -function addGetHookIf( conditionFn, hookFn ) { - - // Define the hook, we'll check on the first run if it's really needed. - return { - get: function() { - if ( conditionFn() ) { - - // Hook not needed (or it's not possible to use it due - // to missing dependency), remove it. - delete this.get; - return; - } - - // Hook needed; redefine it so that the support test is not executed again. - return ( this.get = hookFn ).apply( this, arguments ); - } - }; -} - - -var cssPrefixes = [ "Webkit", "Moz", "ms" ], - emptyStyle = document.createElement( "div" ).style, - vendorProps = {}; - -// Return a vendor-prefixed property or undefined -function vendorPropName( name ) { - - // Check for vendor prefixed names - var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), - i = cssPrefixes.length; - - while ( i-- ) { - name = cssPrefixes[ i ] + capName; - if ( name in emptyStyle ) { - return name; - } - } -} - -// Return a potentially-mapped jQuery.cssProps or vendor prefixed property -function finalPropName( name ) { - var final = jQuery.cssProps[ name ] || vendorProps[ name ]; - - if ( final ) { - return final; - } - if ( name in emptyStyle ) { - return name; - } - return vendorProps[ name ] = vendorPropName( name ) || name; -} - - -var - - // Swappable if display is none or starts with table - // except "table", "table-cell", or "table-caption" - // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display - rdisplayswap = /^(none|table(?!-c[ea]).+)/, - rcustomProp = /^--/, - cssShow = { position: "absolute", visibility: "hidden", display: "block" }, - cssNormalTransform = { - letterSpacing: "0", - fontWeight: "400" - }; - -function setPositiveNumber( _elem, value, subtract ) { - - // Any relative (+/-) values have already been - // normalized at this point - var matches = rcssNum.exec( value ); - return matches ? - - // Guard against undefined "subtract", e.g., when used as in cssHooks - Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : - value; -} - -function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { - var i = dimension === "width" ? 1 : 0, - extra = 0, - delta = 0; - - // Adjustment may not be necessary - if ( box === ( isBorderBox ? "border" : "content" ) ) { - return 0; - } - - for ( ; i < 4; i += 2 ) { - - // Both box models exclude margin - if ( box === "margin" ) { - delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); - } - - // If we get here with a content-box, we're seeking "padding" or "border" or "margin" - if ( !isBorderBox ) { - - // Add padding - delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); - - // For "border" or "margin", add border - if ( box !== "padding" ) { - delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); - - // But still keep track of it otherwise - } else { - extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); - } - - // If we get here with a border-box (content + padding + border), we're seeking "content" or - // "padding" or "margin" - } else { - - // For "content", subtract padding - if ( box === "content" ) { - delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); - } - - // For "content" or "padding", subtract border - if ( box !== "margin" ) { - delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); - } - } - } - - // Account for positive content-box scroll gutter when requested by providing computedVal - if ( !isBorderBox && computedVal >= 0 ) { - - // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border - // Assuming integer scroll gutter, subtract the rest and round down - delta += Math.max( 0, Math.ceil( - elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - - computedVal - - delta - - extra - - 0.5 - - // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter - // Use an explicit zero to avoid NaN (gh-3964) - ) ) || 0; - } - - return delta; -} - -function getWidthOrHeight( elem, dimension, extra ) { - - // Start with computed style - var styles = getStyles( elem ), - - // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). - // Fake content-box until we know it's needed to know the true value. - boxSizingNeeded = !support.boxSizingReliable() || extra, - isBorderBox = boxSizingNeeded && - jQuery.css( elem, "boxSizing", false, styles ) === "border-box", - valueIsBorderBox = isBorderBox, - - val = curCSS( elem, dimension, styles ), - offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); - - // Support: Firefox <=54 - // Return a confounding non-pixel value or feign ignorance, as appropriate. - if ( rnumnonpx.test( val ) ) { - if ( !extra ) { - return val; - } - val = "auto"; - } - - - // Support: IE 9 - 11 only - // Use offsetWidth/offsetHeight for when box sizing is unreliable. - // In those cases, the computed value can be trusted to be border-box. - if ( ( !support.boxSizingReliable() && isBorderBox || - - // Support: IE 10 - 11+, Edge 15 - 18+ - // IE/Edge misreport `getComputedStyle` of table rows with width/height - // set in CSS while `offset*` properties report correct values. - // Interestingly, in some cases IE 9 doesn't suffer from this issue. - !support.reliableTrDimensions() && nodeName( elem, "tr" ) || - - // Fall back to offsetWidth/offsetHeight when value is "auto" - // This happens for inline elements with no explicit setting (gh-3571) - val === "auto" || - - // Support: Android <=4.1 - 4.3 only - // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) - !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && - - // Make sure the element is visible & connected - elem.getClientRects().length ) { - - isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; - - // Where available, offsetWidth/offsetHeight approximate border box dimensions. - // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the - // retrieved value as a content box dimension. - valueIsBorderBox = offsetProp in elem; - if ( valueIsBorderBox ) { - val = elem[ offsetProp ]; - } - } - - // Normalize "" and auto - val = parseFloat( val ) || 0; - - // Adjust for the element's box model - return ( val + - boxModelAdjustment( - elem, - dimension, - extra || ( isBorderBox ? "border" : "content" ), - valueIsBorderBox, - styles, - - // Provide the current computed size to request scroll gutter calculation (gh-3589) - val - ) - ) + "px"; -} - -jQuery.extend( { - - // Add in style property hooks for overriding the default - // behavior of getting and setting a style property - cssHooks: { - opacity: { - get: function( elem, computed ) { - if ( computed ) { - - // We should always get a number back from opacity - var ret = curCSS( elem, "opacity" ); - return ret === "" ? "1" : ret; - } - } - } - }, - - // Don't automatically add "px" to these possibly-unitless properties - cssNumber: { - "animationIterationCount": true, - "columnCount": true, - "fillOpacity": true, - "flexGrow": true, - "flexShrink": true, - "fontWeight": true, - "gridArea": true, - "gridColumn": true, - "gridColumnEnd": true, - "gridColumnStart": true, - "gridRow": true, - "gridRowEnd": true, - "gridRowStart": true, - "lineHeight": true, - "opacity": true, - "order": true, - "orphans": true, - "widows": true, - "zIndex": true, - "zoom": true - }, - - // Add in properties whose names you wish to fix before - // setting or getting the value - cssProps: {}, - - // Get and set the style property on a DOM Node - style: function( elem, name, value, extra ) { - - // Don't set styles on text and comment nodes - if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { - return; - } - - // Make sure that we're working with the right name - var ret, type, hooks, - origName = camelCase( name ), - isCustomProp = rcustomProp.test( name ), - style = elem.style; - - // Make sure that we're working with the right name. We don't - // want to query the value if it is a CSS custom property - // since they are user-defined. - if ( !isCustomProp ) { - name = finalPropName( origName ); - } - - // Gets hook for the prefixed version, then unprefixed version - hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; - - // Check if we're setting a value - if ( value !== undefined ) { - type = typeof value; - - // Convert "+=" or "-=" to relative numbers (#7345) - if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { - value = adjustCSS( elem, name, ret ); - - // Fixes bug #9237 - type = "number"; - } - - // Make sure that null and NaN values aren't set (#7116) - if ( value == null || value !== value ) { - return; - } - - // If a number was passed in, add the unit (except for certain CSS properties) - // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append - // "px" to a few hardcoded values. - if ( type === "number" && !isCustomProp ) { - value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); - } - - // background-* props affect original clone's values - if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { - style[ name ] = "inherit"; - } - - // If a hook was provided, use that value, otherwise just set the specified value - if ( !hooks || !( "set" in hooks ) || - ( value = hooks.set( elem, value, extra ) ) !== undefined ) { - - if ( isCustomProp ) { - style.setProperty( name, value ); - } else { - style[ name ] = value; - } - } - - } else { - - // If a hook was provided get the non-computed value from there - if ( hooks && "get" in hooks && - ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { - - return ret; - } - - // Otherwise just get the value from the style object - return style[ name ]; - } - }, - - css: function( elem, name, extra, styles ) { - var val, num, hooks, - origName = camelCase( name ), - isCustomProp = rcustomProp.test( name ); - - // Make sure that we're working with the right name. We don't - // want to modify the value if it is a CSS custom property - // since they are user-defined. - if ( !isCustomProp ) { - name = finalPropName( origName ); - } - - // Try prefixed name followed by the unprefixed name - hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; - - // If a hook was provided get the computed value from there - if ( hooks && "get" in hooks ) { - val = hooks.get( elem, true, extra ); - } - - // Otherwise, if a way to get the computed value exists, use that - if ( val === undefined ) { - val = curCSS( elem, name, styles ); - } - - // Convert "normal" to computed value - if ( val === "normal" && name in cssNormalTransform ) { - val = cssNormalTransform[ name ]; - } - - // Make numeric if forced or a qualifier was provided and val looks numeric - if ( extra === "" || extra ) { - num = parseFloat( val ); - return extra === true || isFinite( num ) ? num || 0 : val; - } - - return val; - } -} ); - -jQuery.each( [ "height", "width" ], function( _i, dimension ) { - jQuery.cssHooks[ dimension ] = { - get: function( elem, computed, extra ) { - if ( computed ) { - - // Certain elements can have dimension info if we invisibly show them - // but it must have a current display style that would benefit - return rdisplayswap.test( jQuery.css( elem, "display" ) ) && - - // Support: Safari 8+ - // Table columns in Safari have non-zero offsetWidth & zero - // getBoundingClientRect().width unless display is changed. - // Support: IE <=11 only - // Running getBoundingClientRect on a disconnected node - // in IE throws an error. - ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? - swap( elem, cssShow, function() { - return getWidthOrHeight( elem, dimension, extra ); - } ) : - getWidthOrHeight( elem, dimension, extra ); - } - }, - - set: function( elem, value, extra ) { - var matches, - styles = getStyles( elem ), - - // Only read styles.position if the test has a chance to fail - // to avoid forcing a reflow. - scrollboxSizeBuggy = !support.scrollboxSize() && - styles.position === "absolute", - - // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) - boxSizingNeeded = scrollboxSizeBuggy || extra, - isBorderBox = boxSizingNeeded && - jQuery.css( elem, "boxSizing", false, styles ) === "border-box", - subtract = extra ? - boxModelAdjustment( - elem, - dimension, - extra, - isBorderBox, - styles - ) : - 0; - - // Account for unreliable border-box dimensions by comparing offset* to computed and - // faking a content-box to get border and padding (gh-3699) - if ( isBorderBox && scrollboxSizeBuggy ) { - subtract -= Math.ceil( - elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - - parseFloat( styles[ dimension ] ) - - boxModelAdjustment( elem, dimension, "border", false, styles ) - - 0.5 - ); - } - - // Convert to pixels if value adjustment is needed - if ( subtract && ( matches = rcssNum.exec( value ) ) && - ( matches[ 3 ] || "px" ) !== "px" ) { - - elem.style[ dimension ] = value; - value = jQuery.css( elem, dimension ); - } - - return setPositiveNumber( elem, value, subtract ); - } - }; -} ); - -jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, - function( elem, computed ) { - if ( computed ) { - return ( parseFloat( curCSS( elem, "marginLeft" ) ) || - elem.getBoundingClientRect().left - - swap( elem, { marginLeft: 0 }, function() { - return elem.getBoundingClientRect().left; - } ) - ) + "px"; - } - } -); - -// These hooks are used by animate to expand properties -jQuery.each( { - margin: "", - padding: "", - border: "Width" -}, function( prefix, suffix ) { - jQuery.cssHooks[ prefix + suffix ] = { - expand: function( value ) { - var i = 0, - expanded = {}, - - // Assumes a single number if not a string - parts = typeof value === "string" ? value.split( " " ) : [ value ]; - - for ( ; i < 4; i++ ) { - expanded[ prefix + cssExpand[ i ] + suffix ] = - parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; - } - - return expanded; - } - }; - - if ( prefix !== "margin" ) { - jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; - } -} ); - -jQuery.fn.extend( { - css: function( name, value ) { - return access( this, function( elem, name, value ) { - var styles, len, - map = {}, - i = 0; - - if ( Array.isArray( name ) ) { - styles = getStyles( elem ); - len = name.length; - - for ( ; i < len; i++ ) { - map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); - } - - return map; - } - - return value !== undefined ? - jQuery.style( elem, name, value ) : - jQuery.css( elem, name ); - }, name, value, arguments.length > 1 ); - } -} ); - - -function Tween( elem, options, prop, end, easing ) { - return new Tween.prototype.init( elem, options, prop, end, easing ); -} -jQuery.Tween = Tween; - -Tween.prototype = { - constructor: Tween, - init: function( elem, options, prop, end, easing, unit ) { - this.elem = elem; - this.prop = prop; - this.easing = easing || jQuery.easing._default; - this.options = options; - this.start = this.now = this.cur(); - this.end = end; - this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); - }, - cur: function() { - var hooks = Tween.propHooks[ this.prop ]; - - return hooks && hooks.get ? - hooks.get( this ) : - Tween.propHooks._default.get( this ); - }, - run: function( percent ) { - var eased, - hooks = Tween.propHooks[ this.prop ]; - - if ( this.options.duration ) { - this.pos = eased = jQuery.easing[ this.easing ]( - percent, this.options.duration * percent, 0, 1, this.options.duration - ); - } else { - this.pos = eased = percent; - } - this.now = ( this.end - this.start ) * eased + this.start; - - if ( this.options.step ) { - this.options.step.call( this.elem, this.now, this ); - } - - if ( hooks && hooks.set ) { - hooks.set( this ); - } else { - Tween.propHooks._default.set( this ); - } - return this; - } -}; - -Tween.prototype.init.prototype = Tween.prototype; - -Tween.propHooks = { - _default: { - get: function( tween ) { - var result; - - // Use a property on the element directly when it is not a DOM element, - // or when there is no matching style property that exists. - if ( tween.elem.nodeType !== 1 || - tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { - return tween.elem[ tween.prop ]; - } - - // Passing an empty string as a 3rd parameter to .css will automatically - // attempt a parseFloat and fallback to a string if the parse fails. - // Simple values such as "10px" are parsed to Float; - // complex values such as "rotate(1rad)" are returned as-is. - result = jQuery.css( tween.elem, tween.prop, "" ); - - // Empty strings, null, undefined and "auto" are converted to 0. - return !result || result === "auto" ? 0 : result; - }, - set: function( tween ) { - - // Use step hook for back compat. - // Use cssHook if its there. - // Use .style if available and use plain properties where available. - if ( jQuery.fx.step[ tween.prop ] ) { - jQuery.fx.step[ tween.prop ]( tween ); - } else if ( tween.elem.nodeType === 1 && ( - jQuery.cssHooks[ tween.prop ] || - tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { - jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); - } else { - tween.elem[ tween.prop ] = tween.now; - } - } - } -}; - -// Support: IE <=9 only -// Panic based approach to setting things on disconnected nodes -Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { - set: function( tween ) { - if ( tween.elem.nodeType && tween.elem.parentNode ) { - tween.elem[ tween.prop ] = tween.now; - } - } -}; - -jQuery.easing = { - linear: function( p ) { - return p; - }, - swing: function( p ) { - return 0.5 - Math.cos( p * Math.PI ) / 2; - }, - _default: "swing" -}; - -jQuery.fx = Tween.prototype.init; - -// Back compat <1.8 extension point -jQuery.fx.step = {}; - - - - -var - fxNow, inProgress, - rfxtypes = /^(?:toggle|show|hide)$/, - rrun = /queueHooks$/; - -function schedule() { - if ( inProgress ) { - if ( document.hidden === false && window.requestAnimationFrame ) { - window.requestAnimationFrame( schedule ); - } else { - window.setTimeout( schedule, jQuery.fx.interval ); - } - - jQuery.fx.tick(); - } -} - -// Animations created synchronously will run synchronously -function createFxNow() { - window.setTimeout( function() { - fxNow = undefined; - } ); - return ( fxNow = Date.now() ); -} - -// Generate parameters to create a standard animation -function genFx( type, includeWidth ) { - var which, - i = 0, - attrs = { height: type }; - - // If we include width, step value is 1 to do all cssExpand values, - // otherwise step value is 2 to skip over Left and Right - includeWidth = includeWidth ? 1 : 0; - for ( ; i < 4; i += 2 - includeWidth ) { - which = cssExpand[ i ]; - attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; - } - - if ( includeWidth ) { - attrs.opacity = attrs.width = type; - } - - return attrs; -} - -function createTween( value, prop, animation ) { - var tween, - collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), - index = 0, - length = collection.length; - for ( ; index < length; index++ ) { - if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { - - // We're done with this property - return tween; - } - } -} - -function defaultPrefilter( elem, props, opts ) { - var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, - isBox = "width" in props || "height" in props, - anim = this, - orig = {}, - style = elem.style, - hidden = elem.nodeType && isHiddenWithinTree( elem ), - dataShow = dataPriv.get( elem, "fxshow" ); - - // Queue-skipping animations hijack the fx hooks - if ( !opts.queue ) { - hooks = jQuery._queueHooks( elem, "fx" ); - if ( hooks.unqueued == null ) { - hooks.unqueued = 0; - oldfire = hooks.empty.fire; - hooks.empty.fire = function() { - if ( !hooks.unqueued ) { - oldfire(); - } - }; - } - hooks.unqueued++; - - anim.always( function() { - - // Ensure the complete handler is called before this completes - anim.always( function() { - hooks.unqueued--; - if ( !jQuery.queue( elem, "fx" ).length ) { - hooks.empty.fire(); - } - } ); - } ); - } - - // Detect show/hide animations - for ( prop in props ) { - value = props[ prop ]; - if ( rfxtypes.test( value ) ) { - delete props[ prop ]; - toggle = toggle || value === "toggle"; - if ( value === ( hidden ? "hide" : "show" ) ) { - - // Pretend to be hidden if this is a "show" and - // there is still data from a stopped show/hide - if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { - hidden = true; - - // Ignore all other no-op show/hide data - } else { - continue; - } - } - orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); - } - } - - // Bail out if this is a no-op like .hide().hide() - propTween = !jQuery.isEmptyObject( props ); - if ( !propTween && jQuery.isEmptyObject( orig ) ) { - return; - } - - // Restrict "overflow" and "display" styles during box animations - if ( isBox && elem.nodeType === 1 ) { - - // Support: IE <=9 - 11, Edge 12 - 15 - // Record all 3 overflow attributes because IE does not infer the shorthand - // from identically-valued overflowX and overflowY and Edge just mirrors - // the overflowX value there. - opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; - - // Identify a display type, preferring old show/hide data over the CSS cascade - restoreDisplay = dataShow && dataShow.display; - if ( restoreDisplay == null ) { - restoreDisplay = dataPriv.get( elem, "display" ); - } - display = jQuery.css( elem, "display" ); - if ( display === "none" ) { - if ( restoreDisplay ) { - display = restoreDisplay; - } else { - - // Get nonempty value(s) by temporarily forcing visibility - showHide( [ elem ], true ); - restoreDisplay = elem.style.display || restoreDisplay; - display = jQuery.css( elem, "display" ); - showHide( [ elem ] ); - } - } - - // Animate inline elements as inline-block - if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { - if ( jQuery.css( elem, "float" ) === "none" ) { - - // Restore the original display value at the end of pure show/hide animations - if ( !propTween ) { - anim.done( function() { - style.display = restoreDisplay; - } ); - if ( restoreDisplay == null ) { - display = style.display; - restoreDisplay = display === "none" ? "" : display; - } - } - style.display = "inline-block"; - } - } - } - - if ( opts.overflow ) { - style.overflow = "hidden"; - anim.always( function() { - style.overflow = opts.overflow[ 0 ]; - style.overflowX = opts.overflow[ 1 ]; - style.overflowY = opts.overflow[ 2 ]; - } ); - } - - // Implement show/hide animations - propTween = false; - for ( prop in orig ) { - - // General show/hide setup for this element animation - if ( !propTween ) { - if ( dataShow ) { - if ( "hidden" in dataShow ) { - hidden = dataShow.hidden; - } - } else { - dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); - } - - // Store hidden/visible for toggle so `.stop().toggle()` "reverses" - if ( toggle ) { - dataShow.hidden = !hidden; - } - - // Show elements before animating them - if ( hidden ) { - showHide( [ elem ], true ); - } - - /* eslint-disable no-loop-func */ - - anim.done( function() { - - /* eslint-enable no-loop-func */ - - // The final step of a "hide" animation is actually hiding the element - if ( !hidden ) { - showHide( [ elem ] ); - } - dataPriv.remove( elem, "fxshow" ); - for ( prop in orig ) { - jQuery.style( elem, prop, orig[ prop ] ); - } - } ); - } - - // Per-property setup - propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); - if ( !( prop in dataShow ) ) { - dataShow[ prop ] = propTween.start; - if ( hidden ) { - propTween.end = propTween.start; - propTween.start = 0; - } - } - } -} - -function propFilter( props, specialEasing ) { - var index, name, easing, value, hooks; - - // camelCase, specialEasing and expand cssHook pass - for ( index in props ) { - name = camelCase( index ); - easing = specialEasing[ name ]; - value = props[ index ]; - if ( Array.isArray( value ) ) { - easing = value[ 1 ]; - value = props[ index ] = value[ 0 ]; - } - - if ( index !== name ) { - props[ name ] = value; - delete props[ index ]; - } - - hooks = jQuery.cssHooks[ name ]; - if ( hooks && "expand" in hooks ) { - value = hooks.expand( value ); - delete props[ name ]; - - // Not quite $.extend, this won't overwrite existing keys. - // Reusing 'index' because we have the correct "name" - for ( index in value ) { - if ( !( index in props ) ) { - props[ index ] = value[ index ]; - specialEasing[ index ] = easing; - } - } - } else { - specialEasing[ name ] = easing; - } - } -} - -function Animation( elem, properties, options ) { - var result, - stopped, - index = 0, - length = Animation.prefilters.length, - deferred = jQuery.Deferred().always( function() { - - // Don't match elem in the :animated selector - delete tick.elem; - } ), - tick = function() { - if ( stopped ) { - return false; - } - var currentTime = fxNow || createFxNow(), - remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), - - // Support: Android 2.3 only - // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) - temp = remaining / animation.duration || 0, - percent = 1 - temp, - index = 0, - length = animation.tweens.length; - - for ( ; index < length; index++ ) { - animation.tweens[ index ].run( percent ); - } - - deferred.notifyWith( elem, [ animation, percent, remaining ] ); - - // If there's more to do, yield - if ( percent < 1 && length ) { - return remaining; - } - - // If this was an empty animation, synthesize a final progress notification - if ( !length ) { - deferred.notifyWith( elem, [ animation, 1, 0 ] ); - } - - // Resolve the animation and report its conclusion - deferred.resolveWith( elem, [ animation ] ); - return false; - }, - animation = deferred.promise( { - elem: elem, - props: jQuery.extend( {}, properties ), - opts: jQuery.extend( true, { - specialEasing: {}, - easing: jQuery.easing._default - }, options ), - originalProperties: properties, - originalOptions: options, - startTime: fxNow || createFxNow(), - duration: options.duration, - tweens: [], - createTween: function( prop, end ) { - var tween = jQuery.Tween( elem, animation.opts, prop, end, - animation.opts.specialEasing[ prop ] || animation.opts.easing ); - animation.tweens.push( tween ); - return tween; - }, - stop: function( gotoEnd ) { - var index = 0, - - // If we are going to the end, we want to run all the tweens - // otherwise we skip this part - length = gotoEnd ? animation.tweens.length : 0; - if ( stopped ) { - return this; - } - stopped = true; - for ( ; index < length; index++ ) { - animation.tweens[ index ].run( 1 ); - } - - // Resolve when we played the last frame; otherwise, reject - if ( gotoEnd ) { - deferred.notifyWith( elem, [ animation, 1, 0 ] ); - deferred.resolveWith( elem, [ animation, gotoEnd ] ); - } else { - deferred.rejectWith( elem, [ animation, gotoEnd ] ); - } - return this; - } - } ), - props = animation.props; - - propFilter( props, animation.opts.specialEasing ); - - for ( ; index < length; index++ ) { - result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); - if ( result ) { - if ( isFunction( result.stop ) ) { - jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = - result.stop.bind( result ); - } - return result; - } - } - - jQuery.map( props, createTween, animation ); - - if ( isFunction( animation.opts.start ) ) { - animation.opts.start.call( elem, animation ); - } - - // Attach callbacks from options - animation - .progress( animation.opts.progress ) - .done( animation.opts.done, animation.opts.complete ) - .fail( animation.opts.fail ) - .always( animation.opts.always ); - - jQuery.fx.timer( - jQuery.extend( tick, { - elem: elem, - anim: animation, - queue: animation.opts.queue - } ) - ); - - return animation; -} - -jQuery.Animation = jQuery.extend( Animation, { - - tweeners: { - "*": [ function( prop, value ) { - var tween = this.createTween( prop, value ); - adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); - return tween; - } ] - }, - - tweener: function( props, callback ) { - if ( isFunction( props ) ) { - callback = props; - props = [ "*" ]; - } else { - props = props.match( rnothtmlwhite ); - } - - var prop, - index = 0, - length = props.length; - - for ( ; index < length; index++ ) { - prop = props[ index ]; - Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; - Animation.tweeners[ prop ].unshift( callback ); - } - }, - - prefilters: [ defaultPrefilter ], - - prefilter: function( callback, prepend ) { - if ( prepend ) { - Animation.prefilters.unshift( callback ); - } else { - Animation.prefilters.push( callback ); - } - } -} ); - -jQuery.speed = function( speed, easing, fn ) { - var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { - complete: fn || !fn && easing || - isFunction( speed ) && speed, - duration: speed, - easing: fn && easing || easing && !isFunction( easing ) && easing - }; - - // Go to the end state if fx are off - if ( jQuery.fx.off ) { - opt.duration = 0; - - } else { - if ( typeof opt.duration !== "number" ) { - if ( opt.duration in jQuery.fx.speeds ) { - opt.duration = jQuery.fx.speeds[ opt.duration ]; - - } else { - opt.duration = jQuery.fx.speeds._default; - } - } - } - - // Normalize opt.queue - true/undefined/null -> "fx" - if ( opt.queue == null || opt.queue === true ) { - opt.queue = "fx"; - } - - // Queueing - opt.old = opt.complete; - - opt.complete = function() { - if ( isFunction( opt.old ) ) { - opt.old.call( this ); - } - - if ( opt.queue ) { - jQuery.dequeue( this, opt.queue ); - } - }; - - return opt; -}; - -jQuery.fn.extend( { - fadeTo: function( speed, to, easing, callback ) { - - // Show any hidden elements after setting opacity to 0 - return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() - - // Animate to the value specified - .end().animate( { opacity: to }, speed, easing, callback ); - }, - animate: function( prop, speed, easing, callback ) { - var empty = jQuery.isEmptyObject( prop ), - optall = jQuery.speed( speed, easing, callback ), - doAnimation = function() { - - // Operate on a copy of prop so per-property easing won't be lost - var anim = Animation( this, jQuery.extend( {}, prop ), optall ); - - // Empty animations, or finishing resolves immediately - if ( empty || dataPriv.get( this, "finish" ) ) { - anim.stop( true ); - } - }; - - doAnimation.finish = doAnimation; - - return empty || optall.queue === false ? - this.each( doAnimation ) : - this.queue( optall.queue, doAnimation ); - }, - stop: function( type, clearQueue, gotoEnd ) { - var stopQueue = function( hooks ) { - var stop = hooks.stop; - delete hooks.stop; - stop( gotoEnd ); - }; - - if ( typeof type !== "string" ) { - gotoEnd = clearQueue; - clearQueue = type; - type = undefined; - } - if ( clearQueue ) { - this.queue( type || "fx", [] ); - } - - return this.each( function() { - var dequeue = true, - index = type != null && type + "queueHooks", - timers = jQuery.timers, - data = dataPriv.get( this ); - - if ( index ) { - if ( data[ index ] && data[ index ].stop ) { - stopQueue( data[ index ] ); - } - } else { - for ( index in data ) { - if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { - stopQueue( data[ index ] ); - } - } - } - - for ( index = timers.length; index--; ) { - if ( timers[ index ].elem === this && - ( type == null || timers[ index ].queue === type ) ) { - - timers[ index ].anim.stop( gotoEnd ); - dequeue = false; - timers.splice( index, 1 ); - } - } - - // Start the next in the queue if the last step wasn't forced. - // Timers currently will call their complete callbacks, which - // will dequeue but only if they were gotoEnd. - if ( dequeue || !gotoEnd ) { - jQuery.dequeue( this, type ); - } - } ); - }, - finish: function( type ) { - if ( type !== false ) { - type = type || "fx"; - } - return this.each( function() { - var index, - data = dataPriv.get( this ), - queue = data[ type + "queue" ], - hooks = data[ type + "queueHooks" ], - timers = jQuery.timers, - length = queue ? queue.length : 0; - - // Enable finishing flag on private data - data.finish = true; - - // Empty the queue first - jQuery.queue( this, type, [] ); - - if ( hooks && hooks.stop ) { - hooks.stop.call( this, true ); - } - - // Look for any active animations, and finish them - for ( index = timers.length; index--; ) { - if ( timers[ index ].elem === this && timers[ index ].queue === type ) { - timers[ index ].anim.stop( true ); - timers.splice( index, 1 ); - } - } - - // Look for any animations in the old queue and finish them - for ( index = 0; index < length; index++ ) { - if ( queue[ index ] && queue[ index ].finish ) { - queue[ index ].finish.call( this ); - } - } - - // Turn off finishing flag - delete data.finish; - } ); - } -} ); - -jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { - var cssFn = jQuery.fn[ name ]; - jQuery.fn[ name ] = function( speed, easing, callback ) { - return speed == null || typeof speed === "boolean" ? - cssFn.apply( this, arguments ) : - this.animate( genFx( name, true ), speed, easing, callback ); - }; -} ); - -// Generate shortcuts for custom animations -jQuery.each( { - slideDown: genFx( "show" ), - slideUp: genFx( "hide" ), - slideToggle: genFx( "toggle" ), - fadeIn: { opacity: "show" }, - fadeOut: { opacity: "hide" }, - fadeToggle: { opacity: "toggle" } -}, function( name, props ) { - jQuery.fn[ name ] = function( speed, easing, callback ) { - return this.animate( props, speed, easing, callback ); - }; -} ); - -jQuery.timers = []; -jQuery.fx.tick = function() { - var timer, - i = 0, - timers = jQuery.timers; - - fxNow = Date.now(); - - for ( ; i < timers.length; i++ ) { - timer = timers[ i ]; - - // Run the timer and safely remove it when done (allowing for external removal) - if ( !timer() && timers[ i ] === timer ) { - timers.splice( i--, 1 ); - } - } - - if ( !timers.length ) { - jQuery.fx.stop(); - } - fxNow = undefined; -}; - -jQuery.fx.timer = function( timer ) { - jQuery.timers.push( timer ); - jQuery.fx.start(); -}; - -jQuery.fx.interval = 13; -jQuery.fx.start = function() { - if ( inProgress ) { - return; - } - - inProgress = true; - schedule(); -}; - -jQuery.fx.stop = function() { - inProgress = null; -}; - -jQuery.fx.speeds = { - slow: 600, - fast: 200, - - // Default speed - _default: 400 -}; - - -// Based off of the plugin by Clint Helfers, with permission. -// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ -jQuery.fn.delay = function( time, type ) { - time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; - type = type || "fx"; - - return this.queue( type, function( next, hooks ) { - var timeout = window.setTimeout( next, time ); - hooks.stop = function() { - window.clearTimeout( timeout ); - }; - } ); -}; - - -( function() { - var input = document.createElement( "input" ), - select = document.createElement( "select" ), - opt = select.appendChild( document.createElement( "option" ) ); - - input.type = "checkbox"; - - // Support: Android <=4.3 only - // Default value for a checkbox should be "on" - support.checkOn = input.value !== ""; - - // Support: IE <=11 only - // Must access selectedIndex to make default options select - support.optSelected = opt.selected; - - // Support: IE <=11 only - // An input loses its value after becoming a radio - input = document.createElement( "input" ); - input.value = "t"; - input.type = "radio"; - support.radioValue = input.value === "t"; -} )(); - - -var boolHook, - attrHandle = jQuery.expr.attrHandle; - -jQuery.fn.extend( { - attr: function( name, value ) { - return access( this, jQuery.attr, name, value, arguments.length > 1 ); - }, - - removeAttr: function( name ) { - return this.each( function() { - jQuery.removeAttr( this, name ); - } ); - } -} ); - -jQuery.extend( { - attr: function( elem, name, value ) { - var ret, hooks, - nType = elem.nodeType; - - // Don't get/set attributes on text, comment and attribute nodes - if ( nType === 3 || nType === 8 || nType === 2 ) { - return; - } - - // Fallback to prop when attributes are not supported - if ( typeof elem.getAttribute === "undefined" ) { - return jQuery.prop( elem, name, value ); - } - - // Attribute hooks are determined by the lowercase version - // Grab necessary hook if one is defined - if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { - hooks = jQuery.attrHooks[ name.toLowerCase() ] || - ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); - } - - if ( value !== undefined ) { - if ( value === null ) { - jQuery.removeAttr( elem, name ); - return; - } - - if ( hooks && "set" in hooks && - ( ret = hooks.set( elem, value, name ) ) !== undefined ) { - return ret; - } - - elem.setAttribute( name, value + "" ); - return value; - } - - if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { - return ret; - } - - ret = jQuery.find.attr( elem, name ); - - // Non-existent attributes return null, we normalize to undefined - return ret == null ? undefined : ret; - }, - - attrHooks: { - type: { - set: function( elem, value ) { - if ( !support.radioValue && value === "radio" && - nodeName( elem, "input" ) ) { - var val = elem.value; - elem.setAttribute( "type", value ); - if ( val ) { - elem.value = val; - } - return value; - } - } - } - }, - - removeAttr: function( elem, value ) { - var name, - i = 0, - - // Attribute names can contain non-HTML whitespace characters - // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 - attrNames = value && value.match( rnothtmlwhite ); - - if ( attrNames && elem.nodeType === 1 ) { - while ( ( name = attrNames[ i++ ] ) ) { - elem.removeAttribute( name ); - } - } - } -} ); - -// Hooks for boolean attributes -boolHook = { - set: function( elem, value, name ) { - if ( value === false ) { - - // Remove boolean attributes when set to false - jQuery.removeAttr( elem, name ); - } else { - elem.setAttribute( name, name ); - } - return name; - } -}; - -jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { - var getter = attrHandle[ name ] || jQuery.find.attr; - - attrHandle[ name ] = function( elem, name, isXML ) { - var ret, handle, - lowercaseName = name.toLowerCase(); - - if ( !isXML ) { - - // Avoid an infinite loop by temporarily removing this function from the getter - handle = attrHandle[ lowercaseName ]; - attrHandle[ lowercaseName ] = ret; - ret = getter( elem, name, isXML ) != null ? - lowercaseName : - null; - attrHandle[ lowercaseName ] = handle; - } - return ret; - }; -} ); - - - - -var rfocusable = /^(?:input|select|textarea|button)$/i, - rclickable = /^(?:a|area)$/i; - -jQuery.fn.extend( { - prop: function( name, value ) { - return access( this, jQuery.prop, name, value, arguments.length > 1 ); - }, - - removeProp: function( name ) { - return this.each( function() { - delete this[ jQuery.propFix[ name ] || name ]; - } ); - } -} ); - -jQuery.extend( { - prop: function( elem, name, value ) { - var ret, hooks, - nType = elem.nodeType; - - // Don't get/set properties on text, comment and attribute nodes - if ( nType === 3 || nType === 8 || nType === 2 ) { - return; - } - - if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { - - // Fix name and attach hooks - name = jQuery.propFix[ name ] || name; - hooks = jQuery.propHooks[ name ]; - } - - if ( value !== undefined ) { - if ( hooks && "set" in hooks && - ( ret = hooks.set( elem, value, name ) ) !== undefined ) { - return ret; - } - - return ( elem[ name ] = value ); - } - - if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { - return ret; - } - - return elem[ name ]; - }, - - propHooks: { - tabIndex: { - get: function( elem ) { - - // Support: IE <=9 - 11 only - // elem.tabIndex doesn't always return the - // correct value when it hasn't been explicitly set - // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ - // Use proper attribute retrieval(#12072) - var tabindex = jQuery.find.attr( elem, "tabindex" ); - - if ( tabindex ) { - return parseInt( tabindex, 10 ); - } - - if ( - rfocusable.test( elem.nodeName ) || - rclickable.test( elem.nodeName ) && - elem.href - ) { - return 0; - } - - return -1; - } - } - }, - - propFix: { - "for": "htmlFor", - "class": "className" - } -} ); - -// Support: IE <=11 only -// Accessing the selectedIndex property -// forces the browser to respect setting selected -// on the option -// The getter ensures a default option is selected -// when in an optgroup -// eslint rule "no-unused-expressions" is disabled for this code -// since it considers such accessions noop -if ( !support.optSelected ) { - jQuery.propHooks.selected = { - get: function( elem ) { - - /* eslint no-unused-expressions: "off" */ - - var parent = elem.parentNode; - if ( parent && parent.parentNode ) { - parent.parentNode.selectedIndex; - } - return null; - }, - set: function( elem ) { - - /* eslint no-unused-expressions: "off" */ - - var parent = elem.parentNode; - if ( parent ) { - parent.selectedIndex; - - if ( parent.parentNode ) { - parent.parentNode.selectedIndex; - } - } - } - }; -} - -jQuery.each( [ - "tabIndex", - "readOnly", - "maxLength", - "cellSpacing", - "cellPadding", - "rowSpan", - "colSpan", - "useMap", - "frameBorder", - "contentEditable" -], function() { - jQuery.propFix[ this.toLowerCase() ] = this; -} ); - - - - - // Strip and collapse whitespace according to HTML spec - // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace - function stripAndCollapse( value ) { - var tokens = value.match( rnothtmlwhite ) || []; - return tokens.join( " " ); - } - - -function getClass( elem ) { - return elem.getAttribute && elem.getAttribute( "class" ) || ""; -} - -function classesToArray( value ) { - if ( Array.isArray( value ) ) { - return value; - } - if ( typeof value === "string" ) { - return value.match( rnothtmlwhite ) || []; - } - return []; -} - -jQuery.fn.extend( { - addClass: function( value ) { - var classes, elem, cur, curValue, clazz, j, finalValue, - i = 0; - - if ( isFunction( value ) ) { - return this.each( function( j ) { - jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); - } ); - } - - classes = classesToArray( value ); - - if ( classes.length ) { - while ( ( elem = this[ i++ ] ) ) { - curValue = getClass( elem ); - cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); - - if ( cur ) { - j = 0; - while ( ( clazz = classes[ j++ ] ) ) { - if ( cur.indexOf( " " + clazz + " " ) < 0 ) { - cur += clazz + " "; - } - } - - // Only assign if different to avoid unneeded rendering. - finalValue = stripAndCollapse( cur ); - if ( curValue !== finalValue ) { - elem.setAttribute( "class", finalValue ); - } - } - } - } - - return this; - }, - - removeClass: function( value ) { - var classes, elem, cur, curValue, clazz, j, finalValue, - i = 0; - - if ( isFunction( value ) ) { - return this.each( function( j ) { - jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); - } ); - } - - if ( !arguments.length ) { - return this.attr( "class", "" ); - } - - classes = classesToArray( value ); - - if ( classes.length ) { - while ( ( elem = this[ i++ ] ) ) { - curValue = getClass( elem ); - - // This expression is here for better compressibility (see addClass) - cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); - - if ( cur ) { - j = 0; - while ( ( clazz = classes[ j++ ] ) ) { - - // Remove *all* instances - while ( cur.indexOf( " " + clazz + " " ) > -1 ) { - cur = cur.replace( " " + clazz + " ", " " ); - } - } - - // Only assign if different to avoid unneeded rendering. - finalValue = stripAndCollapse( cur ); - if ( curValue !== finalValue ) { - elem.setAttribute( "class", finalValue ); - } - } - } - } - - return this; - }, - - toggleClass: function( value, stateVal ) { - var type = typeof value, - isValidValue = type === "string" || Array.isArray( value ); - - if ( typeof stateVal === "boolean" && isValidValue ) { - return stateVal ? this.addClass( value ) : this.removeClass( value ); - } - - if ( isFunction( value ) ) { - return this.each( function( i ) { - jQuery( this ).toggleClass( - value.call( this, i, getClass( this ), stateVal ), - stateVal - ); - } ); - } - - return this.each( function() { - var className, i, self, classNames; - - if ( isValidValue ) { - - // Toggle individual class names - i = 0; - self = jQuery( this ); - classNames = classesToArray( value ); - - while ( ( className = classNames[ i++ ] ) ) { - - // Check each className given, space separated list - if ( self.hasClass( className ) ) { - self.removeClass( className ); - } else { - self.addClass( className ); - } - } - - // Toggle whole class name - } else if ( value === undefined || type === "boolean" ) { - className = getClass( this ); - if ( className ) { - - // Store className if set - dataPriv.set( this, "__className__", className ); - } - - // If the element has a class name or if we're passed `false`, - // then remove the whole classname (if there was one, the above saved it). - // Otherwise bring back whatever was previously saved (if anything), - // falling back to the empty string if nothing was stored. - if ( this.setAttribute ) { - this.setAttribute( "class", - className || value === false ? - "" : - dataPriv.get( this, "__className__" ) || "" - ); - } - } - } ); - }, - - hasClass: function( selector ) { - var className, elem, - i = 0; - - className = " " + selector + " "; - while ( ( elem = this[ i++ ] ) ) { - if ( elem.nodeType === 1 && - ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { - return true; - } - } - - return false; - } -} ); - - - - -var rreturn = /\r/g; - -jQuery.fn.extend( { - val: function( value ) { - var hooks, ret, valueIsFunction, - elem = this[ 0 ]; - - if ( !arguments.length ) { - if ( elem ) { - hooks = jQuery.valHooks[ elem.type ] || - jQuery.valHooks[ elem.nodeName.toLowerCase() ]; - - if ( hooks && - "get" in hooks && - ( ret = hooks.get( elem, "value" ) ) !== undefined - ) { - return ret; - } - - ret = elem.value; - - // Handle most common string cases - if ( typeof ret === "string" ) { - return ret.replace( rreturn, "" ); - } - - // Handle cases where value is null/undef or number - return ret == null ? "" : ret; - } - - return; - } - - valueIsFunction = isFunction( value ); - - return this.each( function( i ) { - var val; - - if ( this.nodeType !== 1 ) { - return; - } - - if ( valueIsFunction ) { - val = value.call( this, i, jQuery( this ).val() ); - } else { - val = value; - } - - // Treat null/undefined as ""; convert numbers to string - if ( val == null ) { - val = ""; - - } else if ( typeof val === "number" ) { - val += ""; - - } else if ( Array.isArray( val ) ) { - val = jQuery.map( val, function( value ) { - return value == null ? "" : value + ""; - } ); - } - - hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; - - // If set returns undefined, fall back to normal setting - if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { - this.value = val; - } - } ); - } -} ); - -jQuery.extend( { - valHooks: { - option: { - get: function( elem ) { - - var val = jQuery.find.attr( elem, "value" ); - return val != null ? - val : - - // Support: IE <=10 - 11 only - // option.text throws exceptions (#14686, #14858) - // Strip and collapse whitespace - // https://html.spec.whatwg.org/#strip-and-collapse-whitespace - stripAndCollapse( jQuery.text( elem ) ); - } - }, - select: { - get: function( elem ) { - var value, option, i, - options = elem.options, - index = elem.selectedIndex, - one = elem.type === "select-one", - values = one ? null : [], - max = one ? index + 1 : options.length; - - if ( index < 0 ) { - i = max; - - } else { - i = one ? index : 0; - } - - // Loop through all the selected options - for ( ; i < max; i++ ) { - option = options[ i ]; - - // Support: IE <=9 only - // IE8-9 doesn't update selected after form reset (#2551) - if ( ( option.selected || i === index ) && - - // Don't return options that are disabled or in a disabled optgroup - !option.disabled && - ( !option.parentNode.disabled || - !nodeName( option.parentNode, "optgroup" ) ) ) { - - // Get the specific value for the option - value = jQuery( option ).val(); - - // We don't need an array for one selects - if ( one ) { - return value; - } - - // Multi-Selects return an array - values.push( value ); - } - } - - return values; - }, - - set: function( elem, value ) { - var optionSet, option, - options = elem.options, - values = jQuery.makeArray( value ), - i = options.length; - - while ( i-- ) { - option = options[ i ]; - - /* eslint-disable no-cond-assign */ - - if ( option.selected = - jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 - ) { - optionSet = true; - } - - /* eslint-enable no-cond-assign */ - } - - // Force browsers to behave consistently when non-matching value is set - if ( !optionSet ) { - elem.selectedIndex = -1; - } - return values; - } - } - } -} ); - -// Radios and checkboxes getter/setter -jQuery.each( [ "radio", "checkbox" ], function() { - jQuery.valHooks[ this ] = { - set: function( elem, value ) { - if ( Array.isArray( value ) ) { - return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); - } - } - }; - if ( !support.checkOn ) { - jQuery.valHooks[ this ].get = function( elem ) { - return elem.getAttribute( "value" ) === null ? "on" : elem.value; - }; - } -} ); - - - - -// Return jQuery for attributes-only inclusion - - -support.focusin = "onfocusin" in window; - - -var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, - stopPropagationCallback = function( e ) { - e.stopPropagation(); - }; - -jQuery.extend( jQuery.event, { - - trigger: function( event, data, elem, onlyHandlers ) { - - var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, - eventPath = [ elem || document ], - type = hasOwn.call( event, "type" ) ? event.type : event, - namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; - - cur = lastElement = tmp = elem = elem || document; - - // Don't do events on text and comment nodes - if ( elem.nodeType === 3 || elem.nodeType === 8 ) { - return; - } - - // focus/blur morphs to focusin/out; ensure we're not firing them right now - if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { - return; - } - - if ( type.indexOf( "." ) > -1 ) { - - // Namespaced trigger; create a regexp to match event type in handle() - namespaces = type.split( "." ); - type = namespaces.shift(); - namespaces.sort(); - } - ontype = type.indexOf( ":" ) < 0 && "on" + type; - - // Caller can pass in a jQuery.Event object, Object, or just an event type string - event = event[ jQuery.expando ] ? - event : - new jQuery.Event( type, typeof event === "object" && event ); - - // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) - event.isTrigger = onlyHandlers ? 2 : 3; - event.namespace = namespaces.join( "." ); - event.rnamespace = event.namespace ? - new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : - null; - - // Clean up the event in case it is being reused - event.result = undefined; - if ( !event.target ) { - event.target = elem; - } - - // Clone any incoming data and prepend the event, creating the handler arg list - data = data == null ? - [ event ] : - jQuery.makeArray( data, [ event ] ); - - // Allow special events to draw outside the lines - special = jQuery.event.special[ type ] || {}; - if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { - return; - } - - // Determine event propagation path in advance, per W3C events spec (#9951) - // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) - if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { - - bubbleType = special.delegateType || type; - if ( !rfocusMorph.test( bubbleType + type ) ) { - cur = cur.parentNode; - } - for ( ; cur; cur = cur.parentNode ) { - eventPath.push( cur ); - tmp = cur; - } - - // Only add window if we got to document (e.g., not plain obj or detached DOM) - if ( tmp === ( elem.ownerDocument || document ) ) { - eventPath.push( tmp.defaultView || tmp.parentWindow || window ); - } - } - - // Fire handlers on the event path - i = 0; - while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { - lastElement = cur; - event.type = i > 1 ? - bubbleType : - special.bindType || type; - - // jQuery handler - handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && - dataPriv.get( cur, "handle" ); - if ( handle ) { - handle.apply( cur, data ); - } - - // Native handler - handle = ontype && cur[ ontype ]; - if ( handle && handle.apply && acceptData( cur ) ) { - event.result = handle.apply( cur, data ); - if ( event.result === false ) { - event.preventDefault(); - } - } - } - event.type = type; - - // If nobody prevented the default action, do it now - if ( !onlyHandlers && !event.isDefaultPrevented() ) { - - if ( ( !special._default || - special._default.apply( eventPath.pop(), data ) === false ) && - acceptData( elem ) ) { - - // Call a native DOM method on the target with the same name as the event. - // Don't do default actions on window, that's where global variables be (#6170) - if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { - - // Don't re-trigger an onFOO event when we call its FOO() method - tmp = elem[ ontype ]; - - if ( tmp ) { - elem[ ontype ] = null; - } - - // Prevent re-triggering of the same event, since we already bubbled it above - jQuery.event.triggered = type; - - if ( event.isPropagationStopped() ) { - lastElement.addEventListener( type, stopPropagationCallback ); - } - - elem[ type ](); - - if ( event.isPropagationStopped() ) { - lastElement.removeEventListener( type, stopPropagationCallback ); - } - - jQuery.event.triggered = undefined; - - if ( tmp ) { - elem[ ontype ] = tmp; - } - } - } - } - - return event.result; - }, - - // Piggyback on a donor event to simulate a different one - // Used only for `focus(in | out)` events - simulate: function( type, elem, event ) { - var e = jQuery.extend( - new jQuery.Event(), - event, - { - type: type, - isSimulated: true - } - ); - - jQuery.event.trigger( e, null, elem ); - } - -} ); - -jQuery.fn.extend( { - - trigger: function( type, data ) { - return this.each( function() { - jQuery.event.trigger( type, data, this ); - } ); - }, - triggerHandler: function( type, data ) { - var elem = this[ 0 ]; - if ( elem ) { - return jQuery.event.trigger( type, data, elem, true ); - } - } -} ); - - -// Support: Firefox <=44 -// Firefox doesn't have focus(in | out) events -// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 -// -// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 -// focus(in | out) events fire after focus & blur events, -// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order -// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 -if ( !support.focusin ) { - jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { - - // Attach a single capturing handler on the document while someone wants focusin/focusout - var handler = function( event ) { - jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); - }; - - jQuery.event.special[ fix ] = { - setup: function() { - - // Handle: regular nodes (via `this.ownerDocument`), window - // (via `this.document`) & document (via `this`). - var doc = this.ownerDocument || this.document || this, - attaches = dataPriv.access( doc, fix ); - - if ( !attaches ) { - doc.addEventListener( orig, handler, true ); - } - dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); - }, - teardown: function() { - var doc = this.ownerDocument || this.document || this, - attaches = dataPriv.access( doc, fix ) - 1; - - if ( !attaches ) { - doc.removeEventListener( orig, handler, true ); - dataPriv.remove( doc, fix ); - - } else { - dataPriv.access( doc, fix, attaches ); - } - } - }; - } ); -} -var location = window.location; - -var nonce = { guid: Date.now() }; - -var rquery = ( /\?/ ); - - - -// Cross-browser xml parsing -jQuery.parseXML = function( data ) { - var xml, parserErrorElem; - if ( !data || typeof data !== "string" ) { - return null; - } - - // Support: IE 9 - 11 only - // IE throws on parseFromString with invalid input. - try { - xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); - } catch ( e ) {} - - parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; - if ( !xml || parserErrorElem ) { - jQuery.error( "Invalid XML: " + ( - parserErrorElem ? - jQuery.map( parserErrorElem.childNodes, function( el ) { - return el.textContent; - } ).join( "\n" ) : - data - ) ); - } - return xml; -}; - - -var - rbracket = /\[\]$/, - rCRLF = /\r?\n/g, - rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, - rsubmittable = /^(?:input|select|textarea|keygen)/i; - -function buildParams( prefix, obj, traditional, add ) { - var name; - - if ( Array.isArray( obj ) ) { - - // Serialize array item. - jQuery.each( obj, function( i, v ) { - if ( traditional || rbracket.test( prefix ) ) { - - // Treat each array item as a scalar. - add( prefix, v ); - - } else { - - // Item is non-scalar (array or object), encode its numeric index. - buildParams( - prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", - v, - traditional, - add - ); - } - } ); - - } else if ( !traditional && toType( obj ) === "object" ) { - - // Serialize object item. - for ( name in obj ) { - buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); - } - - } else { - - // Serialize scalar item. - add( prefix, obj ); - } -} - -// Serialize an array of form elements or a set of -// key/values into a query string -jQuery.param = function( a, traditional ) { - var prefix, - s = [], - add = function( key, valueOrFunction ) { - - // If value is a function, invoke it and use its return value - var value = isFunction( valueOrFunction ) ? - valueOrFunction() : - valueOrFunction; - - s[ s.length ] = encodeURIComponent( key ) + "=" + - encodeURIComponent( value == null ? "" : value ); - }; - - if ( a == null ) { - return ""; - } - - // If an array was passed in, assume that it is an array of form elements. - if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { - - // Serialize the form elements - jQuery.each( a, function() { - add( this.name, this.value ); - } ); - - } else { - - // If traditional, encode the "old" way (the way 1.3.2 or older - // did it), otherwise encode params recursively. - for ( prefix in a ) { - buildParams( prefix, a[ prefix ], traditional, add ); - } - } - - // Return the resulting serialization - return s.join( "&" ); -}; - -jQuery.fn.extend( { - serialize: function() { - return jQuery.param( this.serializeArray() ); - }, - serializeArray: function() { - return this.map( function() { - - // Can add propHook for "elements" to filter or add form elements - var elements = jQuery.prop( this, "elements" ); - return elements ? jQuery.makeArray( elements ) : this; - } ).filter( function() { - var type = this.type; - - // Use .is( ":disabled" ) so that fieldset[disabled] works - return this.name && !jQuery( this ).is( ":disabled" ) && - rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && - ( this.checked || !rcheckableType.test( type ) ); - } ).map( function( _i, elem ) { - var val = jQuery( this ).val(); - - if ( val == null ) { - return null; - } - - if ( Array.isArray( val ) ) { - return jQuery.map( val, function( val ) { - return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; - } ); - } - - return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; - } ).get(); - } -} ); - - -var - r20 = /%20/g, - rhash = /#.*$/, - rantiCache = /([?&])_=[^&]*/, - rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, - - // #7653, #8125, #8152: local protocol detection - rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, - rnoContent = /^(?:GET|HEAD)$/, - rprotocol = /^\/\//, - - /* Prefilters - * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) - * 2) These are called: - * - BEFORE asking for a transport - * - AFTER param serialization (s.data is a string if s.processData is true) - * 3) key is the dataType - * 4) the catchall symbol "*" can be used - * 5) execution will start with transport dataType and THEN continue down to "*" if needed - */ - prefilters = {}, - - /* Transports bindings - * 1) key is the dataType - * 2) the catchall symbol "*" can be used - * 3) selection will start with transport dataType and THEN go to "*" if needed - */ - transports = {}, - - // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression - allTypes = "*/".concat( "*" ), - - // Anchor tag for parsing the document origin - originAnchor = document.createElement( "a" ); - -originAnchor.href = location.href; - -// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport -function addToPrefiltersOrTransports( structure ) { - - // dataTypeExpression is optional and defaults to "*" - return function( dataTypeExpression, func ) { - - if ( typeof dataTypeExpression !== "string" ) { - func = dataTypeExpression; - dataTypeExpression = "*"; - } - - var dataType, - i = 0, - dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; - - if ( isFunction( func ) ) { - - // For each dataType in the dataTypeExpression - while ( ( dataType = dataTypes[ i++ ] ) ) { - - // Prepend if requested - if ( dataType[ 0 ] === "+" ) { - dataType = dataType.slice( 1 ) || "*"; - ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); - - // Otherwise append - } else { - ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); - } - } - } - }; -} - -// Base inspection function for prefilters and transports -function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { - - var inspected = {}, - seekingTransport = ( structure === transports ); - - function inspect( dataType ) { - var selected; - inspected[ dataType ] = true; - jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { - var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); - if ( typeof dataTypeOrTransport === "string" && - !seekingTransport && !inspected[ dataTypeOrTransport ] ) { - - options.dataTypes.unshift( dataTypeOrTransport ); - inspect( dataTypeOrTransport ); - return false; - } else if ( seekingTransport ) { - return !( selected = dataTypeOrTransport ); - } - } ); - return selected; - } - - return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); -} - -// A special extend for ajax options -// that takes "flat" options (not to be deep extended) -// Fixes #9887 -function ajaxExtend( target, src ) { - var key, deep, - flatOptions = jQuery.ajaxSettings.flatOptions || {}; - - for ( key in src ) { - if ( src[ key ] !== undefined ) { - ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; - } - } - if ( deep ) { - jQuery.extend( true, target, deep ); - } - - return target; -} - -/* Handles responses to an ajax request: - * - finds the right dataType (mediates between content-type and expected dataType) - * - returns the corresponding response - */ -function ajaxHandleResponses( s, jqXHR, responses ) { - - var ct, type, finalDataType, firstDataType, - contents = s.contents, - dataTypes = s.dataTypes; - - // Remove auto dataType and get content-type in the process - while ( dataTypes[ 0 ] === "*" ) { - dataTypes.shift(); - if ( ct === undefined ) { - ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); - } - } - - // Check if we're dealing with a known content-type - if ( ct ) { - for ( type in contents ) { - if ( contents[ type ] && contents[ type ].test( ct ) ) { - dataTypes.unshift( type ); - break; - } - } - } - - // Check to see if we have a response for the expected dataType - if ( dataTypes[ 0 ] in responses ) { - finalDataType = dataTypes[ 0 ]; - } else { - - // Try convertible dataTypes - for ( type in responses ) { - if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { - finalDataType = type; - break; - } - if ( !firstDataType ) { - firstDataType = type; - } - } - - // Or just use first one - finalDataType = finalDataType || firstDataType; - } - - // If we found a dataType - // We add the dataType to the list if needed - // and return the corresponding response - if ( finalDataType ) { - if ( finalDataType !== dataTypes[ 0 ] ) { - dataTypes.unshift( finalDataType ); - } - return responses[ finalDataType ]; - } -} - -/* Chain conversions given the request and the original response - * Also sets the responseXXX fields on the jqXHR instance - */ -function ajaxConvert( s, response, jqXHR, isSuccess ) { - var conv2, current, conv, tmp, prev, - converters = {}, - - // Work with a copy of dataTypes in case we need to modify it for conversion - dataTypes = s.dataTypes.slice(); - - // Create converters map with lowercased keys - if ( dataTypes[ 1 ] ) { - for ( conv in s.converters ) { - converters[ conv.toLowerCase() ] = s.converters[ conv ]; - } - } - - current = dataTypes.shift(); - - // Convert to each sequential dataType - while ( current ) { - - if ( s.responseFields[ current ] ) { - jqXHR[ s.responseFields[ current ] ] = response; - } - - // Apply the dataFilter if provided - if ( !prev && isSuccess && s.dataFilter ) { - response = s.dataFilter( response, s.dataType ); - } - - prev = current; - current = dataTypes.shift(); - - if ( current ) { - - // There's only work to do if current dataType is non-auto - if ( current === "*" ) { - - current = prev; - - // Convert response if prev dataType is non-auto and differs from current - } else if ( prev !== "*" && prev !== current ) { - - // Seek a direct converter - conv = converters[ prev + " " + current ] || converters[ "* " + current ]; - - // If none found, seek a pair - if ( !conv ) { - for ( conv2 in converters ) { - - // If conv2 outputs current - tmp = conv2.split( " " ); - if ( tmp[ 1 ] === current ) { - - // If prev can be converted to accepted input - conv = converters[ prev + " " + tmp[ 0 ] ] || - converters[ "* " + tmp[ 0 ] ]; - if ( conv ) { - - // Condense equivalence converters - if ( conv === true ) { - conv = converters[ conv2 ]; - - // Otherwise, insert the intermediate dataType - } else if ( converters[ conv2 ] !== true ) { - current = tmp[ 0 ]; - dataTypes.unshift( tmp[ 1 ] ); - } - break; - } - } - } - } - - // Apply converter (if not an equivalence) - if ( conv !== true ) { - - // Unless errors are allowed to bubble, catch and return them - if ( conv && s.throws ) { - response = conv( response ); - } else { - try { - response = conv( response ); - } catch ( e ) { - return { - state: "parsererror", - error: conv ? e : "No conversion from " + prev + " to " + current - }; - } - } - } - } - } - } - - return { state: "success", data: response }; -} - -jQuery.extend( { - - // Counter for holding the number of active queries - active: 0, - - // Last-Modified header cache for next request - lastModified: {}, - etag: {}, - - ajaxSettings: { - url: location.href, - type: "GET", - isLocal: rlocalProtocol.test( location.protocol ), - global: true, - processData: true, - async: true, - contentType: "application/x-www-form-urlencoded; charset=UTF-8", - - /* - timeout: 0, - data: null, - dataType: null, - username: null, - password: null, - cache: null, - throws: false, - traditional: false, - headers: {}, - */ - - accepts: { - "*": allTypes, - text: "text/plain", - html: "text/html", - xml: "application/xml, text/xml", - json: "application/json, text/javascript" - }, - - contents: { - xml: /\bxml\b/, - html: /\bhtml/, - json: /\bjson\b/ - }, - - responseFields: { - xml: "responseXML", - text: "responseText", - json: "responseJSON" - }, - - // Data converters - // Keys separate source (or catchall "*") and destination types with a single space - converters: { - - // Convert anything to text - "* text": String, - - // Text to html (true = no transformation) - "text html": true, - - // Evaluate text as a json expression - "text json": JSON.parse, - - // Parse text as xml - "text xml": jQuery.parseXML - }, - - // For options that shouldn't be deep extended: - // you can add your own custom options here if - // and when you create one that shouldn't be - // deep extended (see ajaxExtend) - flatOptions: { - url: true, - context: true - } - }, - - // Creates a full fledged settings object into target - // with both ajaxSettings and settings fields. - // If target is omitted, writes into ajaxSettings. - ajaxSetup: function( target, settings ) { - return settings ? - - // Building a settings object - ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : - - // Extending ajaxSettings - ajaxExtend( jQuery.ajaxSettings, target ); - }, - - ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), - ajaxTransport: addToPrefiltersOrTransports( transports ), - - // Main method - ajax: function( url, options ) { - - // If url is an object, simulate pre-1.5 signature - if ( typeof url === "object" ) { - options = url; - url = undefined; - } - - // Force options to be an object - options = options || {}; - - var transport, - - // URL without anti-cache param - cacheURL, - - // Response headers - responseHeadersString, - responseHeaders, - - // timeout handle - timeoutTimer, - - // Url cleanup var - urlAnchor, - - // Request state (becomes false upon send and true upon completion) - completed, - - // To know if global events are to be dispatched - fireGlobals, - - // Loop variable - i, - - // uncached part of the url - uncached, - - // Create the final options object - s = jQuery.ajaxSetup( {}, options ), - - // Callbacks context - callbackContext = s.context || s, - - // Context for global events is callbackContext if it is a DOM node or jQuery collection - globalEventContext = s.context && - ( callbackContext.nodeType || callbackContext.jquery ) ? - jQuery( callbackContext ) : - jQuery.event, - - // Deferreds - deferred = jQuery.Deferred(), - completeDeferred = jQuery.Callbacks( "once memory" ), - - // Status-dependent callbacks - statusCode = s.statusCode || {}, - - // Headers (they are sent all at once) - requestHeaders = {}, - requestHeadersNames = {}, - - // Default abort message - strAbort = "canceled", - - // Fake xhr - jqXHR = { - readyState: 0, - - // Builds headers hashtable if needed - getResponseHeader: function( key ) { - var match; - if ( completed ) { - if ( !responseHeaders ) { - responseHeaders = {}; - while ( ( match = rheaders.exec( responseHeadersString ) ) ) { - responseHeaders[ match[ 1 ].toLowerCase() + " " ] = - ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) - .concat( match[ 2 ] ); - } - } - match = responseHeaders[ key.toLowerCase() + " " ]; - } - return match == null ? null : match.join( ", " ); - }, - - // Raw string - getAllResponseHeaders: function() { - return completed ? responseHeadersString : null; - }, - - // Caches the header - setRequestHeader: function( name, value ) { - if ( completed == null ) { - name = requestHeadersNames[ name.toLowerCase() ] = - requestHeadersNames[ name.toLowerCase() ] || name; - requestHeaders[ name ] = value; - } - return this; - }, - - // Overrides response content-type header - overrideMimeType: function( type ) { - if ( completed == null ) { - s.mimeType = type; - } - return this; - }, - - // Status-dependent callbacks - statusCode: function( map ) { - var code; - if ( map ) { - if ( completed ) { - - // Execute the appropriate callbacks - jqXHR.always( map[ jqXHR.status ] ); - } else { - - // Lazy-add the new callbacks in a way that preserves old ones - for ( code in map ) { - statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; - } - } - } - return this; - }, - - // Cancel the request - abort: function( statusText ) { - var finalText = statusText || strAbort; - if ( transport ) { - transport.abort( finalText ); - } - done( 0, finalText ); - return this; - } - }; - - // Attach deferreds - deferred.promise( jqXHR ); - - // Add protocol if not provided (prefilters might expect it) - // Handle falsy url in the settings object (#10093: consistency with old signature) - // We also use the url parameter if available - s.url = ( ( url || s.url || location.href ) + "" ) - .replace( rprotocol, location.protocol + "//" ); - - // Alias method option to type as per ticket #12004 - s.type = options.method || options.type || s.method || s.type; - - // Extract dataTypes list - s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; - - // A cross-domain request is in order when the origin doesn't match the current origin. - if ( s.crossDomain == null ) { - urlAnchor = document.createElement( "a" ); - - // Support: IE <=8 - 11, Edge 12 - 15 - // IE throws exception on accessing the href property if url is malformed, - // e.g. http://example.com:80x/ - try { - urlAnchor.href = s.url; - - // Support: IE <=8 - 11 only - // Anchor's host property isn't correctly set when s.url is relative - urlAnchor.href = urlAnchor.href; - s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== - urlAnchor.protocol + "//" + urlAnchor.host; - } catch ( e ) { - - // If there is an error parsing the URL, assume it is crossDomain, - // it can be rejected by the transport if it is invalid - s.crossDomain = true; - } - } - - // Convert data if not already a string - if ( s.data && s.processData && typeof s.data !== "string" ) { - s.data = jQuery.param( s.data, s.traditional ); - } - - // Apply prefilters - inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); - - // If request was aborted inside a prefilter, stop there - if ( completed ) { - return jqXHR; - } - - // We can fire global events as of now if asked to - // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) - fireGlobals = jQuery.event && s.global; - - // Watch for a new set of requests - if ( fireGlobals && jQuery.active++ === 0 ) { - jQuery.event.trigger( "ajaxStart" ); - } - - // Uppercase the type - s.type = s.type.toUpperCase(); - - // Determine if request has content - s.hasContent = !rnoContent.test( s.type ); - - // Save the URL in case we're toying with the If-Modified-Since - // and/or If-None-Match header later on - // Remove hash to simplify url manipulation - cacheURL = s.url.replace( rhash, "" ); - - // More options handling for requests with no content - if ( !s.hasContent ) { - - // Remember the hash so we can put it back - uncached = s.url.slice( cacheURL.length ); - - // If data is available and should be processed, append data to url - if ( s.data && ( s.processData || typeof s.data === "string" ) ) { - cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; - - // #9682: remove data so that it's not used in an eventual retry - delete s.data; - } - - // Add or update anti-cache param if needed - if ( s.cache === false ) { - cacheURL = cacheURL.replace( rantiCache, "$1" ); - uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + - uncached; - } - - // Put hash and anti-cache on the URL that will be requested (gh-1732) - s.url = cacheURL + uncached; - - // Change '%20' to '+' if this is encoded form body content (gh-2658) - } else if ( s.data && s.processData && - ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { - s.data = s.data.replace( r20, "+" ); - } - - // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. - if ( s.ifModified ) { - if ( jQuery.lastModified[ cacheURL ] ) { - jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); - } - if ( jQuery.etag[ cacheURL ] ) { - jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); - } - } - - // Set the correct header, if data is being sent - if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { - jqXHR.setRequestHeader( "Content-Type", s.contentType ); - } - - // Set the Accepts header for the server, depending on the dataType - jqXHR.setRequestHeader( - "Accept", - s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? - s.accepts[ s.dataTypes[ 0 ] ] + - ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : - s.accepts[ "*" ] - ); - - // Check for headers option - for ( i in s.headers ) { - jqXHR.setRequestHeader( i, s.headers[ i ] ); - } - - // Allow custom headers/mimetypes and early abort - if ( s.beforeSend && - ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { - - // Abort if not done already and return - return jqXHR.abort(); - } - - // Aborting is no longer a cancellation - strAbort = "abort"; - - // Install callbacks on deferreds - completeDeferred.add( s.complete ); - jqXHR.done( s.success ); - jqXHR.fail( s.error ); - - // Get transport - transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); - - // If no transport, we auto-abort - if ( !transport ) { - done( -1, "No Transport" ); - } else { - jqXHR.readyState = 1; - - // Send global event - if ( fireGlobals ) { - globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); - } - - // If request was aborted inside ajaxSend, stop there - if ( completed ) { - return jqXHR; - } - - // Timeout - if ( s.async && s.timeout > 0 ) { - timeoutTimer = window.setTimeout( function() { - jqXHR.abort( "timeout" ); - }, s.timeout ); - } - - try { - completed = false; - transport.send( requestHeaders, done ); - } catch ( e ) { - - // Rethrow post-completion exceptions - if ( completed ) { - throw e; - } - - // Propagate others as results - done( -1, e ); - } - } - - // Callback for when everything is done - function done( status, nativeStatusText, responses, headers ) { - var isSuccess, success, error, response, modified, - statusText = nativeStatusText; - - // Ignore repeat invocations - if ( completed ) { - return; - } - - completed = true; - - // Clear timeout if it exists - if ( timeoutTimer ) { - window.clearTimeout( timeoutTimer ); - } - - // Dereference transport for early garbage collection - // (no matter how long the jqXHR object will be used) - transport = undefined; - - // Cache response headers - responseHeadersString = headers || ""; - - // Set readyState - jqXHR.readyState = status > 0 ? 4 : 0; - - // Determine if successful - isSuccess = status >= 200 && status < 300 || status === 304; - - // Get response data - if ( responses ) { - response = ajaxHandleResponses( s, jqXHR, responses ); - } - - // Use a noop converter for missing script but not if jsonp - if ( !isSuccess && - jQuery.inArray( "script", s.dataTypes ) > -1 && - jQuery.inArray( "json", s.dataTypes ) < 0 ) { - s.converters[ "text script" ] = function() {}; - } - - // Convert no matter what (that way responseXXX fields are always set) - response = ajaxConvert( s, response, jqXHR, isSuccess ); - - // If successful, handle type chaining - if ( isSuccess ) { - - // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. - if ( s.ifModified ) { - modified = jqXHR.getResponseHeader( "Last-Modified" ); - if ( modified ) { - jQuery.lastModified[ cacheURL ] = modified; - } - modified = jqXHR.getResponseHeader( "etag" ); - if ( modified ) { - jQuery.etag[ cacheURL ] = modified; - } - } - - // if no content - if ( status === 204 || s.type === "HEAD" ) { - statusText = "nocontent"; - - // if not modified - } else if ( status === 304 ) { - statusText = "notmodified"; - - // If we have data, let's convert it - } else { - statusText = response.state; - success = response.data; - error = response.error; - isSuccess = !error; - } - } else { - - // Extract error from statusText and normalize for non-aborts - error = statusText; - if ( status || !statusText ) { - statusText = "error"; - if ( status < 0 ) { - status = 0; - } - } - } - - // Set data for the fake xhr object - jqXHR.status = status; - jqXHR.statusText = ( nativeStatusText || statusText ) + ""; - - // Success/Error - if ( isSuccess ) { - deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); - } else { - deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); - } - - // Status-dependent callbacks - jqXHR.statusCode( statusCode ); - statusCode = undefined; - - if ( fireGlobals ) { - globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", - [ jqXHR, s, isSuccess ? success : error ] ); - } - - // Complete - completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); - - if ( fireGlobals ) { - globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); - - // Handle the global AJAX counter - if ( !( --jQuery.active ) ) { - jQuery.event.trigger( "ajaxStop" ); - } - } - } - - return jqXHR; - }, - - getJSON: function( url, data, callback ) { - return jQuery.get( url, data, callback, "json" ); - }, - - getScript: function( url, callback ) { - return jQuery.get( url, undefined, callback, "script" ); - } -} ); - -jQuery.each( [ "get", "post" ], function( _i, method ) { - jQuery[ method ] = function( url, data, callback, type ) { - - // Shift arguments if data argument was omitted - if ( isFunction( data ) ) { - type = type || callback; - callback = data; - data = undefined; - } - - // The url can be an options object (which then must have .url) - return jQuery.ajax( jQuery.extend( { - url: url, - type: method, - dataType: type, - data: data, - success: callback - }, jQuery.isPlainObject( url ) && url ) ); - }; -} ); - -jQuery.ajaxPrefilter( function( s ) { - var i; - for ( i in s.headers ) { - if ( i.toLowerCase() === "content-type" ) { - s.contentType = s.headers[ i ] || ""; - } - } -} ); - - -jQuery._evalUrl = function( url, options, doc ) { - return jQuery.ajax( { - url: url, - - // Make this explicit, since user can override this through ajaxSetup (#11264) - type: "GET", - dataType: "script", - cache: true, - async: false, - global: false, - - // Only evaluate the response if it is successful (gh-4126) - // dataFilter is not invoked for failure responses, so using it instead - // of the default converter is kludgy but it works. - converters: { - "text script": function() {} - }, - dataFilter: function( response ) { - jQuery.globalEval( response, options, doc ); - } - } ); -}; - - -jQuery.fn.extend( { - wrapAll: function( html ) { - var wrap; - - if ( this[ 0 ] ) { - if ( isFunction( html ) ) { - html = html.call( this[ 0 ] ); - } - - // The elements to wrap the target around - wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); - - if ( this[ 0 ].parentNode ) { - wrap.insertBefore( this[ 0 ] ); - } - - wrap.map( function() { - var elem = this; - - while ( elem.firstElementChild ) { - elem = elem.firstElementChild; - } - - return elem; - } ).append( this ); - } - - return this; - }, - - wrapInner: function( html ) { - if ( isFunction( html ) ) { - return this.each( function( i ) { - jQuery( this ).wrapInner( html.call( this, i ) ); - } ); - } - - return this.each( function() { - var self = jQuery( this ), - contents = self.contents(); - - if ( contents.length ) { - contents.wrapAll( html ); - - } else { - self.append( html ); - } - } ); - }, - - wrap: function( html ) { - var htmlIsFunction = isFunction( html ); - - return this.each( function( i ) { - jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); - } ); - }, - - unwrap: function( selector ) { - this.parent( selector ).not( "body" ).each( function() { - jQuery( this ).replaceWith( this.childNodes ); - } ); - return this; - } -} ); - - -jQuery.expr.pseudos.hidden = function( elem ) { - return !jQuery.expr.pseudos.visible( elem ); -}; -jQuery.expr.pseudos.visible = function( elem ) { - return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); -}; - - - - -jQuery.ajaxSettings.xhr = function() { - try { - return new window.XMLHttpRequest(); - } catch ( e ) {} -}; - -var xhrSuccessStatus = { - - // File protocol always yields status code 0, assume 200 - 0: 200, - - // Support: IE <=9 only - // #1450: sometimes IE returns 1223 when it should be 204 - 1223: 204 - }, - xhrSupported = jQuery.ajaxSettings.xhr(); - -support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); -support.ajax = xhrSupported = !!xhrSupported; - -jQuery.ajaxTransport( function( options ) { - var callback, errorCallback; - - // Cross domain only allowed if supported through XMLHttpRequest - if ( support.cors || xhrSupported && !options.crossDomain ) { - return { - send: function( headers, complete ) { - var i, - xhr = options.xhr(); - - xhr.open( - options.type, - options.url, - options.async, - options.username, - options.password - ); - - // Apply custom fields if provided - if ( options.xhrFields ) { - for ( i in options.xhrFields ) { - xhr[ i ] = options.xhrFields[ i ]; - } - } - - // Override mime type if needed - if ( options.mimeType && xhr.overrideMimeType ) { - xhr.overrideMimeType( options.mimeType ); - } - - // X-Requested-With header - // For cross-domain requests, seeing as conditions for a preflight are - // akin to a jigsaw puzzle, we simply never set it to be sure. - // (it can always be set on a per-request basis or even using ajaxSetup) - // For same-domain requests, won't change header if already provided. - if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { - headers[ "X-Requested-With" ] = "XMLHttpRequest"; - } - - // Set headers - for ( i in headers ) { - xhr.setRequestHeader( i, headers[ i ] ); - } - - // Callback - callback = function( type ) { - return function() { - if ( callback ) { - callback = errorCallback = xhr.onload = - xhr.onerror = xhr.onabort = xhr.ontimeout = - xhr.onreadystatechange = null; - - if ( type === "abort" ) { - xhr.abort(); - } else if ( type === "error" ) { - - // Support: IE <=9 only - // On a manual native abort, IE9 throws - // errors on any property access that is not readyState - if ( typeof xhr.status !== "number" ) { - complete( 0, "error" ); - } else { - complete( - - // File: protocol always yields status 0; see #8605, #14207 - xhr.status, - xhr.statusText - ); - } - } else { - complete( - xhrSuccessStatus[ xhr.status ] || xhr.status, - xhr.statusText, - - // Support: IE <=9 only - // IE9 has no XHR2 but throws on binary (trac-11426) - // For XHR2 non-text, let the caller handle it (gh-2498) - ( xhr.responseType || "text" ) !== "text" || - typeof xhr.responseText !== "string" ? - { binary: xhr.response } : - { text: xhr.responseText }, - xhr.getAllResponseHeaders() - ); - } - } - }; - }; - - // Listen to events - xhr.onload = callback(); - errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); - - // Support: IE 9 only - // Use onreadystatechange to replace onabort - // to handle uncaught aborts - if ( xhr.onabort !== undefined ) { - xhr.onabort = errorCallback; - } else { - xhr.onreadystatechange = function() { - - // Check readyState before timeout as it changes - if ( xhr.readyState === 4 ) { - - // Allow onerror to be called first, - // but that will not handle a native abort - // Also, save errorCallback to a variable - // as xhr.onerror cannot be accessed - window.setTimeout( function() { - if ( callback ) { - errorCallback(); - } - } ); - } - }; - } - - // Create the abort callback - callback = callback( "abort" ); - - try { - - // Do send the request (this may raise an exception) - xhr.send( options.hasContent && options.data || null ); - } catch ( e ) { - - // #14683: Only rethrow if this hasn't been notified as an error yet - if ( callback ) { - throw e; - } - } - }, - - abort: function() { - if ( callback ) { - callback(); - } - } - }; - } -} ); - - - - -// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) -jQuery.ajaxPrefilter( function( s ) { - if ( s.crossDomain ) { - s.contents.script = false; - } -} ); - -// Install script dataType -jQuery.ajaxSetup( { - accepts: { - script: "text/javascript, application/javascript, " + - "application/ecmascript, application/x-ecmascript" - }, - contents: { - script: /\b(?:java|ecma)script\b/ - }, - converters: { - "text script": function( text ) { - jQuery.globalEval( text ); - return text; - } - } -} ); - -// Handle cache's special case and crossDomain -jQuery.ajaxPrefilter( "script", function( s ) { - if ( s.cache === undefined ) { - s.cache = false; - } - if ( s.crossDomain ) { - s.type = "GET"; - } -} ); - -// Bind script tag hack transport -jQuery.ajaxTransport( "script", function( s ) { - - // This transport only deals with cross domain or forced-by-attrs requests - if ( s.crossDomain || s.scriptAttrs ) { - var script, callback; - return { - send: function( _, complete ) { - script = jQuery( "