how to deal with missing values in ifelse function julia

Question

I am using Julia and I got a dataframe with 42 values, of which 2 are missing.

This values are prices that go from 0.23 to 0.3

I am trying to get a new column that tells if its cheap or expensive by a ifelse statement.

the ifelse should go:

df.x_category=ifelse.(df.x .< mean(df.x),"cheap", "expensive")

but i get the following error:

ERROR: TypeError: non-boolean (Missing) used in boolean context

Is there a way to skip those missing values?

I tried with:

df.x_category=ifelse.(skipmissing(df.x) .< mean(skipmissing(df.x)),"cheap", "expensive")

but get this error:

ERROR: ArgumentError: New columns must have the same length as old columns

I can't just delete missing observations.

How can i make this?

Thanks in advance!

score 4 · Accepted Answer · answered Dec 29 '21 at 02:15

ifelse can handle only 2 values and you need handle 3. Assuming that you have

df = DataFrame(x=rand([0.23,0.3,missing], 10))

than mean(df.x) yields a missing since some of values are missings. You need to do instead mean(skipmissing(df.x))).

Hence the code could be:

julia> map(x -> ismissing(x) ? missing : ifelse(x,"cheap", "expensive"), df.x .< mean(skipmissing(df.x)))
10-element Vector{Union{Missing, String}}:
 missing
 missing
 "cheap"
 missing
 "expensive"
 missing
 missing
 missing
 "cheap"
 "cheap"

Here I have combined ifelse with map for handling the missing value there are other ways but each one will require nesting some conditional function.

score 3 · Answer 2 · answered Dec 29 '21 at 02:25

i would do it with a function that returns cheap, expensive or missing:

using Statistics
data = ifelse.(rand(Bool,100),missing,100*rand(100)) #generator for the data
meandata = mean(skipmissing(data)) #mean of the data

function category_select(x)
  ismissing(x) && return missing  #short-circuit operator
  return ifelse(x<meandata,"cheap","expensive") #parentheses are optional
end

category_select2(x) = ismissing(x) ? missing : (x < meandata ? "cheap" : "expensive)

#broadcast values
x_category = category_selector.(data)
x_category = category_selector2.(data)

now, what is happening? there are two things with the ifelse function:

It evaluates both branches at the same time, so if one branch can error, it will error. take this example:

maybelog(x) = ifelse(x<0,zero(x),log(x)) #ifelse
maybelog2(x) = begin if x<0; zero(x);else;log(x);end #full if expression
maybelog3(x) = x<0 ? zero(x) : log(x) #ternary operator

maybelog fails with x = -1, whereas maybelog2 and maybelog3 does not.

The first argument is always a bool. In your initial expression,the result of df.x .< mean(df.x) can be true, false or missing, so ifelse also fails there.

in your modified expression, the length of skipmissing(df.x) is different than the length of x as the first one doesnt count the missing values present in x, resulting in a smaller vector than the size of your dataframe.

Bogumił Kamiński · Answer 3 · 2023-06-07T10:26:09.367

3

If you are using DataFrames.jl (which it looks like you do) then I recommend you to learn the metapackages that simplify syntax for such scenarios. Here is how you can write your query using DataFrameMacros.jl:

@transform!(df,
            @subset(!ismissing(:x)),
            :x_category = @bycol ifelse.(:x .< mean(:x), "cheap", "expensive"))

This is the simplest approach I think.

edited Jun 07 '23 at 10:26

answered Dec 29 '21 at 09:38

Bogumił Kamiński

66,844
3
80
107

i know i am way too late, but says that @c isn't defined in expression, maybe it's because I am working with a problem a little different – Jorge Paredes Jun 06 '23 at 23:20
1

The syntax has changed. You should use `@bycol` now. – Bogumił Kamiński Jun 07 '23 at 10:26

Andre Wildberg · Answer 4 · 2021-12-29T02:08:35.953

You can try something like this. Using toy data.

First get your string values from ifelse into a vector.
Then prepare the string vector by converting it to a Union of strings and missing to hold missing values.
Finally put the missing values into the vector.

julia> using DataFrames, Random 

julia> vec = ifelse.(df.d[ismissing.(df.d) .== false] .> 0.5,"higher","lower")
40-element Vector{String}:
 "higher"
 "lower"
 "lower"
etc...

julia> vec = convert(Vector{Union{Missing,String}}, vec)
40-element Vector{Union{Missing, String}}

julia> for i in findall(ismissing.(df.d)) insert!(vec, i, missing) end

julia> df.x = vec

julia> df
42×2 DataFrame
 Row │ d                x
     │ Float64?         String?
─────┼──────────────────────────
   1 │       0.533183   higher
   2 │       0.454029   lower
   3 │       0.0176868  lower
   4 │       0.172933   lower
   5 │       0.958926   higher
   6 │       0.973566   higher
   7 │       0.30387    lower
   8 │       0.176909   lower
   9 │       0.956916   higher
  10 │       0.584284   higher
  11 │       0.937466   higher
  12 │ missing          missing
  13 │       0.422956   lower
etc...

Data

julia> Random.seed!(42)
MersenneTwister(42)

julia> data = Random.rand(42)
42-element Vector{Float64}:
 0.5331830160438613
 0.4540291355871424
etc...

julia> data = convert(Vector{Union{Missing,Float64}}, data)
42-element Vector{Union{Missing, Float64}}

julia> data[[12,34]] .= missing
2-element view(::Vector{Union{Missing, Float64}}, [12, 34]) with eltype Union{Missing, Float64}:
 missing
 missing

julia> df = DataFrame(d=data)

how to deal with missing values in ifelse function julia

4 Answers4

Data